SKILL-MIX: a Flexible and Expandable Family of Evaluations for AI Models
Authors: Dingli Yu
Simran Kaur
Arushi Gupta
Jonah Brown-Cohen
Anirudh Goyal
Sanjeev Arora
Venue: ICLR 2024 poster (2024)
Keywords: Large language model
skill evaluation
LLM benchmark
emergence
Primary area: datasets and benchmarks

Abstract: With LLMs shifting their role from statistical modeling of language to serving as general-purpose AI agents, how should LLM evaluations change? Arguably, a key ability of an AI agent is to flexibly combine, as needed, the basic skills it has learned. The capability to combine skills plays an important role in (human) pedagogy and also in a paper on emergence phenomena (Arora & Goyal, 2023).

This work introduces SKILL-MIX, a new evaluation to measure ability to combine skills. Using a list of $N$  skills the evaluator repeatedly picks random subsets of $k$ skills and asks the LLM to produce text combining that subset of skills. Since the number of subsets grows like $N^k$, for even modest $k$ this evaluation will, with high probability, require the LLM to produce text significantly different from any text in the training set. 
The paper develops a methodology for (a) designing and administering such an evaluation, and (b) automatic grading (plus spot-checking by humans) of the results using GPT-4 as well as the open LLaMA-2 70B model. 

Administering a version of SKILL-MIX to popular chatbots gave results that,  while generally in line with prior expectations, contained surprises. Sizeable differences exist among model capabilities that are not captured by their ranking on popular LLM leaderboards ("cramming for the leaderboard"). Furthermore, simple probability calculations indicate that GPT-4's reasonable performance on $k=5$ is suggestive of going beyond "stochastic parrot" behavior (Bender et al., 2021), i.e., it combines skills in ways that it had not seen during training.

We sketch how the methodology can lead to a SKILL-MIX based eco-system of open evaluations for AI capabilities of future models. We maintain a leaderboard of SKILL-MIX at [https://skill-mix.github.io](https://skill-mix.github.io).

Published as a conference paper at ICLR 2024
SKILL-MIX A FLEXIBLE AND EXPANDABLE FAMILY OF EVALUATIONS FOR AI MODELS
Dingli Yuu Simran Kaur" Arushi Gupta'
Jonah Brown-Cohen² Anirudh Goyal Sanjeev Arora!
Princeton Language and Intelligence (PLJ), Princeton University
"Google DeepMind
ABSTRACT
With LLMs shifting their role from statistical modeling of language to serving general-purpose Al agents, how should LLM evaluations change? Arguably, a key ability of an AI agent is to flexibly combine, as needed, the basic skills it has learned. The capability combine skills plays an important role in (human) pedagogy and also in a paper on emergence phenomena (Arora & Goyal 2023).
This work introduces SKILL-MIX, new evaluation measure ability combine skills. Using list of N skills the evaluator repeatedly picks random subsets of & skills and asks the LLM to produce text combining that subset of skills. Since the number of subsets grows like N for even modest i this evaluation will, with high probability, require the LLM to produce text significantly different from any text in the training set. The paper develops methodology for (a) designing and administering such an evaluation, and (b) automatic grading (plus spot-checking by humans) of the results using GPT-4 as well as the open LLaMA-2 70B model.
Administering a version of SKILL-MIX to popular chatbots gave results that, while generally in line with prior expectations, contained surprises. Sizeable differences exist among model capabilities that are not captured by their ranking on popu- lar LLM leaderboards ("cramming for the leaderboard"). Furthermore, simple probability calculations indicate that GPT-4's reasonable performance on A 5 suggestive of going beyond "stochastic parrot" behavior (Bender et al. 2021 i.e., it combines skills in ways that it had not seen during training.
We sketch how the methodology can lead to SKILL-MIX based eco-system of open evaluations for Al capabilities of future models. We maintain a leaderboard of SKILL-MIX at skill -mix. github.io
INTRODUCTION
As LLMs shift roles from mere statistical models of language to fairly general-purpose Al agents, the inadequacy of existing evaluations-eue even those introduced within the past year-har become clear. Yet, leading models routinely score over 90% on many evaluations (OpenAl 2023).
Current evaluations on leaderboards test quantitative, scientific, and academic reasoning suffer from serious limitations. Since they originate from human tests or textbooks, they are vulnerable to training-sel confumination, i.e., examples very similar to the ones on the evaluation ending up in the training corpus of the model (OpenAl 2023 Li, 2023). This is hard measure given the immense size of training corpora and the fact that the corpora are rarely released. The contamination issue especially affects evaluations based upon human exams (whose difficulty is tied to being time-limited and (osed-book) since models are now being trained technical textbooks well as course materials
A variant of the contamination issue is "cramming for the leaderboard." is possible to deliberately train a model on data similar to those used in the leaderboard evaluations. Such datasets are easy to generate from small number of examples using existing strong models. If "cramming" happens
Studies have revealed significant anomalies on GPT-4's performance questions exams created after its release (Narayan & Kapoor).

---

Published as a conference paper at ICLR 2024
Generation
Grading
Input: Write at moes sentences the context of scwing that illustrate all of ponens, red
Input: Could you please help me grade a etudent's response to the following question?
following skills: herring, and mataphor.
Write at most sentences the contaxt of sawing that Ilustrate all of the following skills: modus ponens, rod herring, and metaphor.
Question:
Generating model: If needles the keys cratting melodles, than every perfect shitch would be a note in hermonious symphony; but speaking of symphories, have you ever naticed how the early bird's song sounds just like Mozart?
Student's Answer: If needleee were the keys to crafting melodies, then every perfect stitch would be note hamonnous symphony, but speaking of symphories. have you ever noticed how the early bird's song sounds just like Mozart?
Grading rubric: Correctly illustrates skills (3 points), On topic (1 point), Makes sense point), At most sentances point).
Grading model: would give the student the following score:
point for modus gonens
point for red horring
point for metaphor
0.5 point for baing on topic
point for making seriee
point for the length requirement
Figure 1: Left: Simplified depiction (with simplified prompt) of the generation stage of our eval- uation. The full prompt appears in Appendix C.3.I] The generating model is given a topic (sewing) as well skills (modus ponens, red herring, metaphor), and asked generate text demonstrating the skills. The full prompt contains skill definitions and examples, which can be found in Appendix [C.3.] Right: Simplified depiction (with simplified prompt) of the grading stage of our evaluation. The grading model (not necessarily the same as the generating model) is given the generating model output and grading instructions, and returns pointwise grading. The full grading prompt can be found in Appendix C.3.2
during pre-training, it becomes hard to detect. If it happens during fine-tuning, it may be detectable if it ends up harming general-purpost language skills.
Yet another issue arising from the secrecy of the training corpus is that it is difficult to verify how original the model's text productions truly are. In a recent interview (Hinton & Ng). Hinton suggested that a significant hurdle in current discussions of AI risk is absence of agreement among experts on whether or not models have already gone past "stochastic parrots" behavior (Bender et al. 20211-i.. whether they are able to actually understand the world, or at minimum produce novel thoughts or "behavior" that they did not see in the training corpus.
To sum up, we need evaluations that are: (a) clearly relevant to general- intelligence and language understanding: (b) casy for humans to design and administer, including with academic-level resources; (c) resistant to training-set contamination and "cramming for the leaderboard;" (d) capable of revealing LLM's capability on producing novel text that they did not see in the training corpus (Hinton & Ng), i.e., going beyond "stochastic" parrots behavior (Bender et al., (021); (e) easy to grade at scale (while allowing human spot-checking): (f) easily upgradeable into harder evaluations in the future as models get stronger; and (g) comprehensible at some level for the general public, including with respect to points (c) and (d).
Desiderata for next-generation evaluations:
1.1 SKILL-MIX
Our SKILL-MIX evaluation tests the model's ability to produce sensible text satisfying natural con- straints. It starts with set of N skills (an example is: "use of metaphor") that every LLM could reasonably be expected to have encountered in its training set -say, because each skill has a Wikipedia entry. SKILL-MIX also uses list of T topics that have low, but nonnneglig,,,,, probability in any reasonable training corpus -c.g.. "dueling." " "gardening." " SKILL-MIX (k) consists of randomly picking subset of k skills out of N, and random topic out of 7 topics. The chat agent is then asked to produce a short piece of text (say, 3 sentences) in the context of the selected topic and illustrate all l selected skills. This evaluation can be easily administered using any set of skills and questions and any k. The model is being required to do highly constrained text generation, whose difficulty intuitively increases with 2
TThe authors of this paper took an average of more than 7 minutes answer SKILI-MIX (4). See Table 1. for some examples of text generated by humans vs. GPT-4
2

---

Published as a conference paper at ICLR 2024
Since the evaluation task consists of sampling random k-tuple from distribution, performance can be estimated reliably with a few hundred runs of the model. Thus human grading is feasible with modest budgets. We found that it is just reliable to do auto-grading followed by human spot-checking of the grading. We used two easily-available models GPT-4 (OpenAl 2023 and
LLaMA-2-70B-Chal (Touvron et al. 2023).
SKILL-MIX is easily tailored to a model's capability by adjusting k and in general larger models are able to handle larger (Section [.2). In the future, as models get stronger, the evaluation can be made harder by not only increasing k but also by adding new skills to our list, or by switching to more difficult or specialized skills, or to topics that are rarer or more specialized. It is also possible to create SKILL-MIX evaluations for special domains (c.g. coding or science) and for multi-modal data, although we leave that for future work.
SKILL-MIX has two theoretical inspirations. The first are theories of human pedagogy and learning (Forehand) 2005 Koedinger et al., 2012 2023 Li et al. (013), which have long focused on the ability to combine skills. The other is a recent paper (Arora & Goyal 2023) that gave a theory for how complex skills emerge in LLMs when they are scaled up. The paper assumed that understanding small pieces of text consists of applying k skills out of large list of basic skills. Under some assumptions, the paper showed that scaling up language models leads to improvement in proficiency at applying k' e'-tuples of basic skills, where k increases with scaling. (Roughly speaking, k' doubles when the mode is scaled up 10x.) Its appendix included a few examples of text produced by current chatbots in response requests to combine given set of skills, and that an inspiration for SKII.L-MIXX
Difficulty of SKILL-MIX with increasing k. The intuition that the hardness of SKILL-MIX increases with k can be supported by a simple calculation. Given a list of N skills, there are (N) ways to choose the subset of skills. For N = 1000, this quantity exceeds 10 when k = 4, and 10 when k = 5. Furthermore, the skills and topics are fairly rare in the corpus ee.g. the word "sushi" has a unigram probability of 10-7 in Google n-grams (Google 2012). Thus, the chance that the training corpus contains short piece of text on the chosen topic that exhibits the chosen set of k skills becomes very small even for N = 100 (see Section 6. Such calculations are very relevant to be convinced that a model is going past "stochastic parrots' behavior as well as has resistance to dataset contamination.
Paper Organization. Section 2 describes related prior work. Section 3 laas out design principles for the evaluation: assembling the list of skills and topies, as well as the prompt used to administer Section describes the setup for auto- auto-grading. Section repporss the experimental results. Section 6 gives the calculation for verifying that the model going beyond stochastic parrots behavior. Section
sketches a future framework of several SKILL-MIX evaluations maintained by disinterested research groups, with a secret list of skills and topics.
1.2 KEY FINDINGS
Section SJstion shoows results of administering SKILL-MIX (k) for different & to today's leading models. For proprietary models, the results are mostly in accord with prior expectations. GPT-4 ranks highest, being able to perform reasonably even for k = 5. LLaMA-2-70B-Cha. turns in a good performance, somewhat below GPT-3.5 Turbo, which is in line with general consensus about its capabilities. For most models, k- = 3 already proves too tough.
Evidence of cramming for the leaderboard. Hugging Face's Open LLM leaderboard (Beeching et al. 2023). which is based upon EleutherAl's evaluation harness (Gao et al. 2021). is seen as a proving ground for open LLMs. Many models currently at the top of the leaderboard are LLaMA- 2 derivatives, and are ranked much higher than the corresponding LLaMA-2 model. However, on SKILL-MIX these models perform poorly and worse than LLaMA- 70B BhCttt,, suggestive of cramming that significantly harmed general-purpose text skills (see Section 5 The recent Falcon- 180B-Chat (Almazrouei et al., 2023) also places higher on the leaderboard than L.LaMA--700B- and has been claimed to have capabilities between GPT-3.5-turbe and GPT-4 based upon this ranking. Yet, it fares worse than LLaMA-2-700 live up to claims of being significantly better than the corresponding LLaMA model.
on SKILL-MIX Mistral- B also did not
Detecting saturation of an evaluation. AlpacaEval (Li et al. 2023 is popular evaluation (with an accompanying leaderboard) for text generation. It uses GPT-4 and dataset of prompts to check how often the model's generated output "wins" " against that of Da Vinci003, AlpacaEval presents a good case-study on the difficulties of creating good evaluations. Designed to give small open models
3

---

Published as a conference paper at ICLR 2024
fighting chance in early 2023, it shows signs of saturation mere 6 months later. Even 13B models now win against Da Vinci003 with probability exceeding 90%, and recently Xwin-LM- V (built on the LLaMA-2 base models) climbed to the top. pushing GPT-4, by a hair, to second place. Did L.M capabilities truly progress this much within 6 months? We find that the new champion scores well on SKII.L-MIX, and noticeably better than ALaMA-2-770B 2-B-Chhh,,, but it is handily beaten by GPT-4. In general, most reasonable models now get similar scores on AlpacaEval, but show widely varying performances on SKILL-MIX, which suggests that AlpacaEval has lost its discriminative abilityy SKILL-MIX avoids this shortcoming by evaluating on constrained text generation, and using k to adjust the difficulty.
Demonstrating capabilities beyond "stochastic parrot." Whether or not current models are capable of going past "stochastic parrots" behavior (Bender al. 202T) remains a crucial topic of discussion in the field (Hinton & Ng), and it is unclear what this would mean. In context of SKILL-MIIX we define it to mean: ability to correctly use combinations of skills topic that it had not seen in the training corpus. Our version of SKILL-MIX uses list of skills each of which has Wikipedia entry or listing (and thus known to every LLM). As mentioned, GPT-4 generates correct answers with reasonable probability for SKILL-MIX (A= 5), and we estimate that at least 1/3 of the time this involves text that combines a mix of skills and topics that had not appeared in the training corpus. For k = 6, we estimate the majority of the correct answers generated by GPT-4 involve novel combinations (see fection[]land Appendix for calculations). We are unable to find similar evidence for any other model at this time.
Design principles for SKILL-MIX. We include ablation studies suggesting design principles for new versions of SKILL-MIX : (i) auto grading setup (Section 4 (ii) deducting points for explicitly mentioning skill names in the answer (Appendix C.6) (iii) filterngg out common skills (Appendix C.6p. Our initial experiments involved a set of 101 skills selected from books and Wikipedia. Our calculation in Section 6 suggests that including skills that have higher frequencies in the training corpus makes SKIIII-MIXX easier. We identified 17 skills that appeared with rather high frequency in the common crawl corpus. Omitting them from the evaluation appeared to make it much harder for most models (see Section C.6). This second version is recommended.
2 PRIOR WORK
Arora & Goyal (2023) gives a mathematical model for skill emergence via LLM scaling. (In principle it applies to non-text data as well, since makes almost no assumptions about what "text" or "skills" are.) The key assumption is that pieces of text involve random combinations of skills, and then reductions in cross- cross-entropy (which can happen on an arbitrary subset of text pieces) are shown to imply improvements in both individual skills and combinations of skills. A key implication is that the trained model may show competence on k-tuples of skills even though this k-tuple of skills was not demonstrated in the training set. (Note that individual skills were demonstrated, as were some random k-tuples, but most k-tuples were not demonstrated.) It is important to note that the theory did not need specify what it means "combine" skills.
The goal of Skill-It (Chen et al. 2023) is to select an optimal subset of the training data such that an LLM trained on this subset will perform well on downstream tasks. Chen et al. (2023) utilize the notion of skill ordering to construct this subset. Skill ordering refers to the natural notion that learning "simpler" skills first can make learning "difficult" skills later easier for the learner. Chen et al. (202)) define an ordered skill set as "a collection of skills with directed skills graph that is neither complete nor empty, where an edge from a prerequisite skill to a skill exists if the amount of training it takes to learn the skill can be reduced if the prerequisite skill is also learned."
Skill-It
Skills in Reinforcement Learning Reinforcement learning also has notion of "skills," " which is distinct from the notion that we use (Arora & Goyal| 2002). but bears some similarities to the notion of skill used in Chen et al. (2023). In particular, Pertsch et al. (202T) aims to learn a skill prior, ie., a distribution over skills, such that model trained to develop these skills will later perform well on downstream tasks.
Prior evaluations Prior work has emerged which evaluates LLMs on particular skills, in different sense than we do. Ruis et al. (2023) finds that LLMS do not do well on implicature.
Pedagogy work Another important area where skills have been previously studied is that of human skill learning in pedagogy. Koedinger et al.) (2023) develops a cognitive and statistical model of skill acquisition with the goal of understanding why/if some students learn faster than others. Koedinger
4

---

Published as a conference paper at ICLR 2024
et al. (2012) presents an algorithm to discover cognitive models, which are essentially skill models. Li et al. (2013) uses a computational model of student learning (a simulated student) in order to discover cognitive models (i.e., "learn the skills").
3 DESIGN OF SKILL-MIX
3.1 PICKING THE SKILLS
We obtained set of 101 language skills and a set of 100 topics for SKILL-MIX evaluation. Since the goal of our evaluation is test general-purpose text generation capability rather than the ability of the particular skills and topics. we only release 10 skills and 10 topics randomly sampled from the two sets to avoid potential "cramming" for SKILL-MIX The randomly sampled skills and topics appear in Appendix A
(see Tables and
We curated the topic list by first sampling large list of topies (e.g. using Reddit forums as inspiration), and then narrowing the list down to 100 topics based on the unigram frequency of the topic (and all related synonyms) on Google Ngram viewer (Google 2012). To earn a spot on our list, a topic's 's average unigram had to be around 10 -low (further reducing the likelihood that the model had seen the skills demonstrated in the context of the topic), and yet still ensuring good coverage even in 100B-sized corpora. The list of basic skills was designed to contain language skills which have Wikipedia entry or listing (and thus known to every L.LM), and whose definition the average person could understand. We started with longer list of skills taken from texthooks logical reasoning. rhetoric. theory of mind, (Kelly. 2014; Cummings 2005, Nichols & Stich 2003 Mueller 2014). We tried eliminate skills that either were too specialized and thus difficult apply in the context of the fairly narrow topics already chosen- or difficult to combine with other skills. (Some examples of discarded skills appear in the Appendix C.2) ) For each skill, we created description and an illustrative example of its usage -these were taken from either a textbook or Wikipedia, though occasionally, we modified them to make them clearer or more concise.
3.2 PROCEDURE
Our evaluation is roughly broken down into two parts (see Figure 2}. In the first part, we conduct generation, where a (Student) language model is given a set of k skills and a topic, and asked to generate some text demonstrating the & skills in the context of the provided topie. Once the (Student) language model generates some text, it then must be graded by (possibly different) grading language model (i.e., Grader). A simplified version of the prompts used in the generation and grading stages is depicted in Figure
Models used for generation Our dataset designed to test general skills, and hence many language models may be used in the generation step. However, since the language model must be able to respond to prompt instructions ("generate skills"), we only pick models that have been instruction-tuned. These models include LLaMA-2-7B--O
LLMA-22-1B-CHI LLaMA-2-70B. Chat Touvron et al., 2023), GPT-3.5-turbo. GPT-4 (OpenAl 2023), Falcon- (8B-CBBBaa (Almazrouei
et al.
2023), Xwin- Xwin-LM-70B LM-70B---VOOO V0.1 (Xwin LM Team 2023). - (Mistral AI 2023), Qwen-14B-Chal (Bai et al. (023). Sicerbot-7BB-Che
(TigerResearch) 2023). Note and Qwen- DDwen-14BCCHN 14B-Chat were released only few days before the deadline, and all of them claim to perform better on benchmarks compared to statc-of-the- art models of even larger size.
Team
X Xwin-LMM70BB- 10 Mistrra--B--nntrruet--
Models used for grading We find that some language models are more suitable for grading than others. Some models have difficulty recognizing the presence of skills, even when they are demonstrated correctly. We use 2---MA---7--CCAC and GPT-4 after manually spot-checking grading samples and ensuring they aligned with human grading.
SKILL-MIX (k) consists of picking a random topic and random subset of k skills (with their respective definitions and illustrative examples) from the list described above, and asking the (Student) model to produce short piece of text which illustrates all the k skills in the context of the topic of interest. A simplified example prompt appears in Figure [ but our final prompt includes an additional question that asks the (Student) model to look over and possibly improve its first answer. We find the second answer can be much better than the first one with proper prompt engineering. In addition, we ask the model to separate its answer and explanation with "Answer" and "Explanation." Otherwise. the model may not separate its generated answer from its explanation, hampering the parsing of the answer. We provide more details about the generation prompts in Appendix C.3.1
5

---

Published as a conference paper at ICLR 2024
The authors took the test (6 questions each with lo 4) to assess the feasibility and difficulty level. The average time taken to understand the prompt and type the answer was more than minutes. This is not an easy test for humans!
AUTO-GRADING METHOD
For each prompt sampled from SKILL-MIX (k), the (Student) model's corresponding answer is graded according to the following criteria: (1) the skills are present and used properly in the output; (2) the output on the given topic; (3) the number of sentences in the output within the provided limit, which we set to be k 1; and (4) the output is piece of sensible text. For any of the above subtasks, partial credit can be assigned if the answer partially satisfies the requirement.
The generations by the models were graded using GPT-4 as well as LLaMA-2-700 -h-t,,, and these grades were then spot-checked by the paper authors, In the trial run, we focused on tweaking the method for best results and consistency using a small set of around 20 generations. As usual, the assessments generated by the grading models (especially ILaMA-2) were somewhat sensitive to the prompt. We tweaked the grading prompt by including summarized version of the generation prompt, providing definitions and illustrative examples of the individual & skills, and requesting for the graded output to follow a particular format.
While both models are creditable graders, they, like all current LLMs, were unreliable at simple arithmetic, which is important for calculating total score. We changed the prompt to require the grader to output separate grades for individual components (proper use of skills, good to the topic, and producing sensible text), which subsequently aggregated (see Figure using separate but simple Python scripp]
To require separate grades for individual components, asked GPT-4 to provide rubric-table style grade, whereas for for LLaMA-2, we simply asked the model to include "Point earned: 1 if the requirement is met, and "Point earned: 0" otherwise, for each rubric item in the evaluation. More details, as well as the full grading prompt, appear in Appendix C.3.2
Human Grading: Better? With small test we conducted with five NL.P researchers, we found that human grading is noisy, and human graders might need significant training so they can agree on grading rubric. The standard deviation between the human grading high, and even higher than the difference between the average human grading and GPT-4 grading This also indicates that GPT-4 and LLaMA-2 graders are reasonable graders compared humans. More details of the test can be found in Appendix B
EXPERIMENTAL RESULTS
In this section, we test various instruction-tuned models (including the LLaMA-2 family, GPT family, Falcon- Milcon-180B-6e 80B-Chat, Xwin-LM-70B- 10. 70B-Chat) their performance on SKILL-MIX for various k. For convenience, we use saturation point to denote the value at which a model's score in SKILL-MIX drops off.
Swven-14BCCin and Tigerbot-
From the experimental results, answer the following questions
What differences arise between grading by GPT-4 vs. LLaMA-2?
What is the effect of increasing & on SKILL-MIX performance? What is the saturation point for the instruction-tunel models?
What is the relationship between model scale and saturation point? We particularly interested in answering this question for the family of LLaMA-2 models, since they share the same training set and methodology.
Setup We evaluate various Instruction-tunel models on SKILL-MIX (k) with k = 2, We continue to evaluate model with l (potential also l 6) if it does saturate 4. We use both GPT-4 and LLaMA-2 2-B---------- thtt as Grader.
We adjusted the point awarded for meeting the number of sentences requirement hased the ground truth. While adjustments were rare, they were more common for a.L.MA-2 than GPT-4. 'The difference between the average human grading and I.I.MA-2 grading is slightly higher than the standard deviation between humans.

---

Published as a conference paper at ICLR 2024
Table 1: Performance of various instruction- student (generating) models on SKILL-MIX (k)
graded by GPT-4. Ratio of Full Marks/Ratio of All Skills/Skill Fraction are reported for each student
model at l = 2, 3, Evaluations on & = 5, 6 are skipped if the Ratio of Full Marks drops below 0.05
with smaller A. Details on prompts can be found in Section
3 See Table 5 for additional metrics.
Student (generator)
k k=2 2
k k=3 3
k= 4
==
k=6 6
LLaMA-2-1 2---A--2-B--hii
.5/.111/37
.00/.01/.25
.00.000/23
-//-
-/--
LLaMA 2-13B--hhha
.221.351.45
.05/.05/.37
.02/.02/.45
-1--
4/-
LLLMA-2-770--- 70t-Chat
.2812281.2
.02/.05/46
.0//01//44
-1--
-1--
GPT-3.5-turbo
53/.601.75
.20125/.59
088/.3//54
.04/.16/.51
-1--
GPT
94/.971.96
.681.70/.88
52/.55/.88
.36/.88/.6
.291.291.84
.05/.16/.40
.00/.057.36
000.04//25
-|--
-1--
Qwen- 14B-Chat
.16/.19/.50
.021.05/.39
,01/.0//.8
-1--
4-1-
Xwin-LM-70B- V0.1
421.58/.67
.221.7/1.3
19//17
.08/.12/.5
-1--
malcon-180B-Cha
.77133/153
.00/.3//44
03/.07/38
-1--
-1--
Tigerbot-70B-Chat
.06/.291.26
.00%.10019
1117.06/15
-1--
-H---
For each SKILL-MIX (k), we evaluate all models on 100 (A: skills, 1 topic) combinations with GPT-4 grading, and 30 combinations with LLaMA-2 grading 5 We provide each specific combination of k skills to the (Student) model three instances (see Figure 2). Each of the three generated texts are also graded three times (to reduce the randomness caused by the Grader), in total creating nine grading results for each (A skills, topic) (see Figure 3)
Metries Each generated text can receive up to + 3 points: point for each correctly illustrated skill, 1 point for sticking to the topic, point for coherence/ / making sense, and 1 point for having most k 1 sentence. Recall that we grade each generated text three times. In each round of grading, we parse each of the criteria individually from the Grader model's output. For each criterion, we then collect the majority vote among the three grading rounds. The voted points are then converted into various metrics of interest (see Appendix C.5). some of which include
la of Full Marks: if all & + 3 points are earned, and 0 otherwise
Ratio of All Skills: 1 & points are awarded for the k skills and at least 2 points are awarded for the remaining criteria, and and 0 otherwise
Skill Fraction: the fraction of points awarded for the l skills all 3 points are awarded for the remaining criteria, and 0 otherwise
We then take the maximum value of the metrics among the generations for a given (A: skill, topic) combination, and average the maximum value across all the combinations.
Differences in Grader Scores From Tables and we clearly observe that L.LMA-22 a more generous grader than GPT-4. We also observe that when LLaMA-2 is used as the grader, it prefers generations outputted by the LLaMA-2 family. For example, across different metrics, GPT-4 than to -.aMA----BB-hhil but LLaMA-2
generally gives a higher score to Destral-BB-Instruet- grader gives a much higher score to ALaMA-2-7B-Chm than Mistral- I for & 2 3. Overall, we found via spot-checking the output that GPT-4 is more accurate and reliable grader than LLaMA-2.
Increasing i degrades SKILII-MIX performance We observe that the ratio of full marks and the ratio of all skills can decrease dramatically when & increases. With the exception of GPT-4, all models saturate on or before k 3 with GPT-4 grading. Amongst the small models, LLaMA-2-7B-Chal and Distral-7B-Instruct--
GPT-3.5-turbo and XWin-LM-70B-
saturate at A = 2. Since LLaMA-2 more generous, the saturation point is usually delayed by for LLaMA-2 grading.
Relationship between model scale and saturation point We find that as capacity increases on LLaMA-2, so does the saturation point. Observe that for --L.MM----7--CC
LLaMA-2-13B-OL
LLaMA-2-70B- -hhat, the saturation points (of GPT-4 grading) are k = 2, 3, and 3, respectively. Additionally, for any fixed l and metric type, higher model capacity corresponds to better score
'Due to computation limits, we only evaluate Falcon- Mecoon-10BB-CC 80B-Chat on 30 combinations with GPT-4 grading. TThis is called "ratio" because the metric is later averaged over the 30 combinations, even though this metric 0 and I for single generation.
7

---

Published as a conference paper at ICLR 2024
amongst the LLaMA-2 model family. However, these observations do not necessarily hold true for models from different families. For example, Falcon- Valcon-180B-oo 180B- Chat has more model parameters than Xwin-LM-70B VO.1, yet the saturation point of Xwin-LM-70B- V0.1 is higher than that of Falcon- Milcon-180B-Che 80B-Chat, and Xwin-LM-70B-VVL also outperforms Falcon- Malcon-180B-ce 88BBChhat across all metrics for k = 3,4.4
deviation from model rankings on popular LLM leaderboards Recent models (i.e., Falcon- 180B-Chat, Xwin-LM-70B-VO.1 Qwen-14B-Chat Mistral-7B--nnstruct-vo. are often introduced with their performance evaluated on AlpacaEval or Hugging Face's ' Open LLM Leaderboard (which contains ARC (Clark et al.
2018). HellaSwag (Zellers et al. 2019), MMLU (Hendryeks et al. 2020),
and TruthfulQA (Lin et al. 2021)), along with a comparison to the LLaMA-2 and GPT families. We find that their superior performance on those evaluations (which is often misaligned with general experience) may not extend to SKILI-MIX
Falcon-180B-CHI and Tigerbot- Bocrbbot-7BB-haar rank higher than LLaMA-2-7000 -LaMMA-2-70B-Ch on Open LLM Leaderboard, but perform worse on SKILL-MIX for both GPT-4 and LLaMA-2 grading. 70gerbat performs even worse than LLaMA-2-13B-ChM
Xwin-LM-70B--O.. takes on first place on AlpacaEval, beating GPT-4. However, Xwin- LM-70B-V0.1 is clearly worse than GPT-4 on SKILL-MIX
on MMLU, HumanEval (Chen et al. 2021 and GSM8K (Cobbe et al. 2021), but performs worse than LLaMA-2-70B-Char for k 2, 2, 4 with both GPT-4 and LLMMA-2 grading.
Qwen- 14B-Chat outperforms --
Mistral-7B-vdl outperforms L-2 13B on all benchmarks that the Mistral Al team (the model after instruction tuning) outperforms LLaMA-2- 13B-Chat on MT-Bench (Zheng et al. 2023). Yet, the situation is reversed on SKILL-MIX.
tested.
6 EVIDENCE FOR GOING BEYOND "STOCHASTIC PARROTS" BEHAVIOR
As mentioned in the introduction, there is interest in whether not current models are capable of going past "stochastic parrots" " behavior. Using SKILL-MIX we are able to give a guarantee: for & = and k 6, GPT-4 is generating correct answers a good fraction of time, and reasonable portion (say more than 1 1/3) of these correct answers contain combinations of skills and topics that have never jointly appeared in the training. We are unable find similar evidence for any other model.
Our claim is verified by upperbounding the average frequency of skills ps, the average frequency of topics Pt, and the number of sentences in the training corpus L. Then in the training corpus, the total number of text piece that contains i of the skills and one of the topics is bounded by p (N (T)T.. ag is the model's s "Ratio of Full Marks" in SKILL-MIX (k), then the number of text pieces that successfully generates with of the skills and one of the topics is at least ag (Z) T. Thus, the model is "beyond stochastic parrots" if a > 3pEP,L, meaning more than one-third of the successful generations contain combinations not seen in the training corpus.
After filtering out skills that have frequency of at least 5%, we estimate that with high probability. Ps < 0.0144, pe < 0.0022 and L < 5 x 10 on RedPajama dataset (Computer 2023). Thus, L < 0.001 for k ===0 6, and pL L < 0.07 for & = 5. Even after filtering out common skills as in Appendix C.6 GPT-4 still has 0g Rs 0.08 and asş A 0.12, which is still "beyond stochastic parrots."
Please Appendix D for more details of the calculation and estimation.
7 BEST PRACTICES FOR SKILL-MIX ECOSYSTEM
SKILL-MIX differs from most existing evaluations in two ways: (1) there is no dataset per instead, by using N skills and T topics, the tasks (prompts) are generated randomly on the fly from (N) possible combinations; and (2) for moderate k; the task not easy for humans to solve, or even to grade; minimum, high-quality human labor is needed.
However, note that a dataset of O(T log T log N) random prompts and corresponding productions already expose (with high probability) the full set of skills and topics, as well as many interesting
"This calculation assumes independence among the occurrence of skills. Verifying k-wise independence hecomes computationally very expensive for large hut we verified it for l 2.
8

---

Published as a conference paper at ICLR 2024
ways to combine them. Preliminary results (in Appendix E suggest that fine-tuning such dataset of synthetic productions can improve scores on MIX. Since the goal of our evaluation is to test general-purpose text generation capability rather than ability on the particular skills and topics, we propose to release only a random subset of 10% of skills and topics.
But, ultimately, needs an evaluation ecosystem. This might consist of independent research groups developing versions of SKILL-MIX (in other words, tests that are chosen randomly from a high number of potential questions) with very different sets of skills and topics. including basic skills related to science, economics, law, etc. Releasing a random subset of, say, 10% of the skills and topics of a new evaluation gives the rest of the world an idea of what it tests -and how it is reasonably distinct from other existing evaluations- while retaining its difficulty. If the research groups are seen as trustworthy, this ecosystem's s continuous assessment of Al capabilities might become important inputs into policy discussions in the future.
Difficulty of grading An open question in the above picture how to grade harder versions of SKILL-MIX in the future. The obvious idea is to use the current champion model for the evaluation (provided the model retains no memory of its past interactions). But, a natural question arises: whether to trust the champion's 's grade for itself. This relates to an interesting debate in pedagogy (Bloom et al.) 1956 Anderson & Krathwohl 2001; Forehand 2005): Which is harder: to ace the exam to grade it well? While the answer may seem obvious in quantitative or scientific fields (i.e., acing is harder), this wasn't 't obvious in other fields. Today it is more broadly accepted that grading is indeed easier) which suggests that the champion can probably grade itself (assuming the model retains no memory of past interactions), Human spot-checking seems advisable. In our experience, GPT-4's grading capabilities for particular k seem be better than its SKILL-MIX score on the same k; we plan to further investigate this relationship in future work. On a related note, we found that human spot-checking becomes much easier and more accurate if the grader includes reasoning for grading decisions.
8 CONCLUSIONS AND TAKEAWAYS
SKILL-MIX is an attempt to evaluate general-purposs language capabilities, specifically, particular sort of compositional generalization. It tests the model's ability to create text on a given topic and with given combination of well-known skills. A key idea that the skills and the topic are chosen randomly from big list, which itself can be expanded in many ways to give new evaluations. Such evaluations may end up requiring the (Student) model to imagine and describe situations that does not exactly match anything seen during training. Grading this evaluation is shown to be possible with GPT-4 (and the authors spot-checked the grading). Human evaluations were sought but found to have significant variance. We will try to reduce this variance in the next run with better rubrics and guidance.
Section 5 showed that the performance of proprietary models on SKILI-MIIX (k) generally accords with popular perceptions of their quality. The results also show (in line with the theory of [Arora & Goyal (223)) that when created by competent teams, larger models achieve a higher saturation point than smaller models (e.g., the three LLaMA-2 models, which shared their training data and methodology). The disappointing performance of Falcon-1 180B-Chat was exception. Several open models show signs of being over-trained for leaderboards at the expense of general-purpor language capabilities ("cramming"). Furthermore, based on our calculations. GPT-4's reasonable performance on SKILL-MIX (k 5) indicates that GPT-4 surpasses "stochastic parrot" behavior (Bender et al.) 2021). Of course, no single evaluation can be considered definitive. In future work, we hope to explore whether there are "cramming" shortcuts to improve performance on SKILL-MIX.
Since current leaderboards show signs of losing their discriminative power, Section 7 sketched a vision for an ecosystem of independent SKILL-MIX evaluations specializing in different sets of skills and topics (most of which would be kept secret) that could provide trusted estimates of model capabilities both as input to public discussions of AI and as deterrent against "cramming for leaderboards". SKILL-MIX scores track our colleagues' subjective assessments of current models.
Soon, many AI models will be multi-modal, and we hope to design multi-modal version of SKILL-MIX for them.
'utt the variance we saw among human graders on our SKIIII-MIX reminds of the difficulty of grading exams where there is no obvious hest answer.

---

Published as a conference paper at ICLR 2024
Acknowledgements:
This work is partly supported by NSF and ONR.
REFERENCES
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Co- jocaru, Maitha Alhammadi, Mazzotta Daniele, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series of language models: Towards open frontier models. 2023.
Lorin W. Anderson and David R. Krathwohl (eds.). Taxonomy for Learning, Teaching, and Assessing. A Revision of Bloom's Taxonomy of Educational Objectives. Allyn & Bacon, New York, edition, December 2001. ISBN 978-0801319037.
Sanjeev Arora and Anirudh Goyal. theory for emergence of complex skills in language models. 2023.
arXiv preprint arXiv:2307.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li. Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang. Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang. Jian Yang. Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. 2023.
Edward Beeching, Clémentine Fourrier, Nathan Habib. Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open Ilm leaderboard. https
spaces http://aaa....ccppppe/////igiiia
1lmpppeee____________________1__ 1m__mmm__eee_e__eeeeeeeelleellle lelleeederbrrrr 2023.
Emily M Bender, Timnit Gebru, Angelina MeMillan-Major and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp. 610-623, 2021.
B. S. Bloom, M. Engelhart, E. J. Furst, W. H. Hill, and D. R. Krathwohl. Taxonomy of educational objectives. The classification of educational goals. Handbook 1: Cognitive domain. Longmans Green, New York, 1956.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03377 2021.
Mayee F Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré. Skill-it! data-driven skills framework for understanding and training language models. arXiv e-prints, pp. arXiv-2307, 2023.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
Together Computer. Redpajama: An open source recipe to reproduce llama training dataset, 2023. URL lattps:/
jama Datt
Louise Cummings. Pragmatics: a multidisciplinaary perspective. In Edinburgh University Press, 2005.
Mary Forehand. Emerging perspectives learning, teaching, and technology. chapter Bloom's taxonomy: Original and revised. 2005.
Leo Gao, Jonathan Tow, Stella Biderman. Sid Black, Anthony DiPofi, Charles Foster, Laurence Gold- ing, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021. URL URLhttps: //do .org /tt///////////// zenado. 5331 628
10

---

Published as a conference paper at ICLR 2024
Google. Google ngram viewer. http://book.ggogg....momggmmaaas 2012. URL http: //books. google. www.nnggam//daaaaaaaa.a..a
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint MrXii:2009.033000 2020.
Hinton and A Ng. A conversation about ai risk. status 667 Stt.L://....22220222222222222220
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 2021.
David Kelly. The art of reasoning. an introduction to critical and logical thinking. In WW Norton & Company Inc, 2014.
Kenneth R Koedinger, Elizabeth A McLaughlin, and John C Stamper. Automated student model improvement. International Educational Data Mining Society, 2012.
Kenneth Koedinger, Paulo F Carvalho, Ran Liu, and Elizabeth A McLaughlin. An astonishing regularity in student learning rate. Proceedings of the National Academy of Sciences, 120(13): c2221311 120, 2023.
Nan Li, Eliane Stampfer, William Cohen, and Kenneth Koedinger. General and efficient cognitive model discovery using a simulated student. Proceedings of the Annual Meeting of the Cognitive Science Society, volume 35, 2013.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of followiinnn models. https:: //tt:s///ttt thub. www.:at.ttttattaaaaaaaaa paca _ecc_ 2023.
Yucheng Li. Estimating contamination via perplexity: Quantifying memorisation in language model evaluation, 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.0999 2021.
Edward Loper and Steven Bird. Nltk: The natural language toolkit. arXiv preprint cs/2005028 2002.
Mistral AI Team. Mistral 7b, 9 2023.
URL https://mistee
sunouncing-mistrall
Erik T Mueller. Commonsense reasoning: an event calculus based approach. Morgan Kaufmann, 2014.
Arvind Narayan and Sayash Kapoor.
Gpt-4 and professional benchmarks: the wrong
answer to the wrong question. URL
Shaun Nichols and Stephen P Stich. Mindreading, an integrated account of pretence, self-awareness, and understanding other's minds. In Oxford University Press, 2003.
OpenAI. Gpt-4 technical report, 2023.
Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pp. 188-204, PMLR, 2021.
Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, and Edward Grefenstette. Large language models are not zero-shot communicators. NeuRIPS, 2023.
Philipp Schmid, Omar Sanseviero, Pedro Cuenca, Leandro von Werra, and Julien Launay. Spread your wings: Falcon 180b is here, 2023. URL httpps://haggingfface. co/ /
fa con-18bl

---

Published as a conference paper at ICLR 2024
TigerResearch. TigerBot: A cutting-edge foundation for your very own llm. https://githu: com hww.Tggggeeeaareeeeeeee.... 2023. Accessed: 2023-09-30.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arX00:2307.009288 2023.
Xwin-LM Team. Xmi 9 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence?, 2019.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Bric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judee with mt-bench and chatbot arena, 2023.
12

---

Published as a conference paper at ICLR 2024
A LIST OF SKILLS AND TOPICS
Here we release a random sample of 10% the skill list and topic list. We do not release the full lists avoid potential "cramming" for SKILL-MIX
Table 10 randomly sampled skills from the 101 skills we used in SKILL-MIX evaluation.
Category
Skill
Definition
Example
reasoning
self serving bias
A cognitive or perceptual process that distorted by the need to maintain and enhance one's self es- teem.
"If I do well on the exam, it's because my academic prowess and hard work. If do poorly, it's because the course was poorly taught, and the exam was poorly proctored."
rhetorical
accident (fallacy)
an informal fallacy and a deductively valid but unsound argument oc- curring in statistical syllogism (an argument based on generaliza- tion) when an exception to a rule of thumb is ig- nored.
Cutting people with knives is a crime. Surgeons cut peo- ple with knives. Surgeons are criminals.
rhetorical complex question (loaded question with implicit as-
A question that is loaded with an implicit assump- tion.
"Why are you lying to me?" is a question that presup- poses you are lying to me. Any answer you give will force you to agree you are lying.
sumption
rhetorical
red herring
Introducing irrelevant points to detract attention from a question.
member of the press asks the president why they voted expand a welfare program. The president responds, "The strength of America is the strength of its communities, and am proud make our communities better places." "All the world's a stage, And all the men and women merely players" is a metaphor because it's a comparison without using "like" "as."
literary
metaphor
a figure of speech that, for rhetorical effect, di- rectly refers to one thing by mentioning another.
logical
spatial reasoning
The capacity to reason about the spatial relation- ships between objects.
The key fit into the box. Us- ing spatial reasoning, one deduce that the width of the key was smaller than the width of the box.
logical
A syllogism that is of modus ponens
"If today is Tuesday, then John will go to work. Today is Tuesday. Therefore, John will go to work."
the form "If P then Q. Hence Q."
13

---

Published as a conference paper at ICLR 2024
logical
statistical gism
syllo-
syllogism that argues, using inductive reason- ing, from a generaliza- tion true for the most part to particular case.
"Almost all people are taller than 26 inches. Gareth is person. Therefore, Gareth is taller than 26 inches."
theory of the mind
emotional self reg- ulation
a complex process that in- volves initiating. inhibit- ing, or modulating one's state behavior in given situation.
Examples of emotional self regulation include meditat- ing, pausing to collect one- self before speaking, and practicing stress manage- ment.
physical knowl- edge
folk physics (com- knowledge physics)
The untrained human per- ception of basic physical phenomena.
"If roll the pen off of the table, it will fall the floor."
Table 3: 10 random sampled topics from 100 topics used in SKILL-MIX evaluation.
Topic
Sewing
Dueling
The Ottoman Empire
Triathlons
Beekeeping
Survivalism
Guerilla warfare
Gardening
Knots
Urbanism
B HUMAN GRADING TEST
We will now describe test we conducted to measure the grading quality of human graders. Our five volunteers were Ph.D. students and Postdocs working in the field of natural language processing and large language models. They were given 5 outputs generated by GPT-4 SKILL-MIX (4). The same grading prompt for machine grading was given human graders, asking them to give each individual point for the criteria. In this case (k 4), each output could receive most points.
For each point, we computed the mean and standard deviation among the human graders and then averaged over all the points (35 in total). The average standard deviation is 0.261. On the other hand, we compare the average of human grading with GPT-4 and LLaMA-2 grading. For each point, we computed the absolute difference between the mean among the humans and machine grading. Then, we took the average over 35 points. We found that the average difference is 0.257 for GPT-4 grading and 0.268 for LLaMA-2 grading. This means if we assume the average of human grading is the ground truth, then human graders and machine graders have similar errors.
We also observe that human graders in general give lower scores for text making sense, probably because the output is usually a shortened version of the model's 's first attempt. In contrast, the machine graders usually award the point for "making sense" for more than 90% of the generated answers.
C EXPERIMENTS
C.I EXPERIMENTAL DETAILS
Our evaluation is roughly broken down into two parts. In the first part, we conduct generation, where language model is given set of & skills and topic and asked generate text demonstrating
14

---

Published as a conference paper at ICLR 2024
the l skills. Once some text has been generated, it then must be graded by (possibly different) language model.
PART 1: GENERATION
PART 2: GRADING
Generate
(A skill topici contanatiooa
f)
Apgrogated
Generated Tixt (Rtund 2)
Gil Aggragated Cinade
Take maxirun score
Select
Generoted Tax (Round
Gel Aggregnted Grade
Skudient Model
Selest
Resort overago over s00rs3
Grador Wodd
Generaied tix jilsand
Gal Cinade
Genorated (Round
Apgroqated
Gemerated ledt (iound 3)
Get Aparagaled Ginade
Figure 2: Illustration of SKILL-MIX (k) pipeline. In our experiments, we use M = 30. For more detailed illustration of grading a single piece of generated text, see Figure
Models used for generation. Our dataset is designed to test general skills, and hence many language models may be used in the generation step. However, since the language model must be able to respond to prompt instructions ("generate k skills"), we pick only models that have been instruction-tunee These models include I.LaMA-2-7B-Chal I.LaM- 1--3B-Chht, LLaMAA2-70B; Chat (Touvron et al. 2023), GPT-3.5-turbo, GPT-4 (OpenAl 2023), Falcon-1800 180B-Cha (Almazrouei
ct al.
2023). Xwin-LM-70B V0.1 (Xwin-LM Team 2023 I--nsrutt...st (Mistral Al
Team
2023), and 14ben- (Bai et al. 2023).
Models used for grading. We find that some language models are more suitable for grading than others. Some models have difficulty recognizing the presence of skills, even when they demonstrated correctly. We use LLLaM-2-77B-CCTT and GPT-4 after manually spot-checking grading samples and ensuring they align with human grading.
Details of model configurations We do not use quantization on any of the models. For the GPT family, we use OpenAI API with default generation configuration and the minimal system prompt "You are a helphul system prompt, 0.7 temperature, 1.0 repetition penalty, and 512 max new tokens. For Falcon- 180B- Chat, we use the prompt format mentioned in the official Huggingface blog of Falcon- 180B -hatt (Schmid al. 2023) and the same parameters as LLaMA-2 family. For Xwin-LM--7B- VO.11 we use the official prompt format (Xwin-IM Team 2023) and the same hyperparameters as those used for the L-MMA-2 family. For -B-sarrtt
assistant. For the LLaMA-2 family, we use 2 A100 GPU and run with no
we access the prompt format with tokenizer wwwlyppphhaaeeee seayyyhhaatttmppeeee.. function and again the same parameters the LLaMA-2 family. For Leven-14B-Char we directly use the model. chat function as mentioned in their official Github repository.
C.2 SKILL CHOOSING PROCESS
The list of zs 100 skills used to draw tuples of & skills was manually curated, and designed to include skills the average person could understand which were common enough that they would appear Wikipedia (and hence in the model's 's pre-training data). We pick skills from standard textbooks on logic (Kelly. 2014), pragmatics (Cummings 2005) and theory of mind (Nichols & Stich 2003). We also pick literary skills from Wikipedia. Not all skills were considered suitable for our dataset. Because we want evaluate the ability of models to compose skills, we eliminate skills that are trivially present in almost any piece of text. Below is an example of a skill that was eliminated from our dataset because it is so common the English language that the model may "accidentally" generate it, thereby falsely appearing as though it is able to combine this skill with other skills.
Skill: Fied Piping
Decinition: A syntax phenomenon whereby a given expression brings along an accompanying phrase when it is movad.
15

---

Published as a conference paper at ICLR 2024
GRADING ROUND
AGGREGATE POINTS
RUBRIC
Rate Markey Dj Fich B) mn ge-33 othewwnee
Connecty Bustroeee SMI
Connecty Bustzene
Connectly Bustrate SMIA
F and (9 otherwiee
Marua serse
Atmost b-7 sorfances
AGGREGATE INDVVDUAL POINTS
RUBAIC ITENS
Scits Fraction
GRADING ROUND
poin
(AW [o
(f otheratio
Comextl ausouees Skil Corrccly Ausooles Ski
Take sum
Elich
Score
Conrecty Businatss
Comestl kastuats Skil
Generated Text
Skill Boore
Correcty Bustzine
Makie most but sentenowe
Toplc:
Miloes seroa sentonces
Apgreseting Oplicee (toaa ningle
Rescaled
A+B
Maxc
ROUND
RUBRiC ITES
Face Nom worth poice
Connecty Buninine 54817
On Tople
seroe
Figure 3: Illustration of obtaining aggregated grade This illustration depicts the process used to grade a single generated piece of text.
16

---

Published as a conference paper at ICLR 2024
Ex: She bought Mchat house''. "hich house"r did she buy? The preceding is an exampla pied piping because the "Nh...'' ! el ause.
"Which"" brings along word "house""
For similar reasons, we also eliminate skills that inherently compose poorly with other skills. An example is given below:
skill: situational irony
rirccouse burning down situatiooaal rony, would expect that puts out Lires Lo buzn.
Finally, some skills were not included due to being hard, even for a human grader, to grade whether skill was present.
Definitions for skills vary between different sources and textbooks. Since the model was pre-trained on Wikipedia, we prefer the Wikipedia definition over other sources when available.
Skill examples were scraped from the internet, but chosen by human evaluators be short and unambiguous.
C.3 PROMPT DESIGN
We experiment with different prompts for both generation and grading. We find that prompts can perform quite differently. We list some examples of prompts we considered below.
C.3.1 GENERATION PROMPTS
We try several different prompts when asking the models generate tuples of skills, and find that prompt selection does influence the quality of the generation. In general, our prompts contain two questions, giving model the chance to look over its first answer and improve Without giving specific instructions about the format, we found hard to parse the model output for grading because the model may not separate the generated answer from its explanation, Hence, we asked the model to separate its answer and explanation with "Answer:" and "Explanation:" Here is our first prompt with the formatting instructions:
Great ingstings! am interested in natural language processing and was wondaring if you could help me generate an example of text that lustrates multiple aki aemans tes or syatax. The example should be single piece of Lext with (num.eentences.atu in the context of (topin) that illustrates all of the following skills: {skills str).
please keep the text short so it can fit in (aum.sentencas.att)) and pleass make sure the concepta found fully atart the explenstion with "Exglamations". Thanks very much!
Please the with YAnswer!"
Thanks very much. Now could d you look it over and shorten your example but make sure it atill illustrates
skills?
Using this prompt, observe that the models do always follow the instructions, especially for the second answer. We also observed that the second answer is sometimes worse than the first generation, partially because some of the skills also removed when shortening the answer.
To overcome the shortages, we present our 8th attempt at the generation prompt below.
interested in natural language processing and wondering if you could help me generata en axemple of text that illustrates multiple skills in semancics 1ines
Greetings!
syntax. The example ahould be minimal natural plece text with up in the context of [topic) that illustrates all of the following skills; (skilla.ste) possible,
2lease keep the text Erem the text.
found Eully sure concepts
reference,
are definitions examples concepts:
(ski is dafs and examples simple)
17

---

Published as a conference paper at ICLR 2024
Please start the minimal natural piece of text with 'Answer:' and start the explanation with Explooattion"".
Thonka very much!
Thanks very much, Could you please look over the minimal natural piece of Lex. and possibly improve shorten (up (numssentencessszehy make sure that the text still illustrates all skills and remaina on topic.
you changes, please
Again, plaase start improved the explanation with "Explanations".
natural piace text 'Anawor!' start
Thanks very nuch!
Although there still exists rare failure cases for some models (demonstrated in Appendix I... the instructions are mostly followed by all models with this version of the prompt. The quality of the second answer is also significantly better when we remove the length constraints on the first answer, and give the model the option of shortening the text (rather than the requirement) for the second answer.
In the later tuning of the prompt, further emphasize the improvement of the second answer, and find the performance of GPT-4 continues to improve. The same prompt also improves Xwin-L.M- 70B-VO 10.1, but makes LLaMA-2 family perform slightly worse k >> 2 (in those cases, the previous prompt). Our final prompt is recorded below.
Greetings! am interested in natural language processing and was wondering if you could help ne generate an example of text that iilustrates multiple skilla semant or syatax. The example should be minimal natural plece of text with up to sew 1ines in the context of [topic) that illustrates all the following skills: (skills.sad) be found fully
elease keep the text short pessible, and make the from the text.
reference,
the definitions and examples
concepti:
mininal natural piece Lext with 'Answers' with "Explanation:'.
explanation
very much!
Thanks very much. Could you please look your and improve it? Please make sure that the new answer batter illustrates all skills and remains on tepic. on top of that, please
new only centains up (nun.sentences.eus).
Again, please scart the Improved minimal natural piece of text with 'Anawers' and start the explanation with 'Explanation''
Thanks very much!
Example of LLaMMA-2770B-Chal generation response
Sure, here''s example of minimal natural piece of text that (llutrates boch emctional self requlation and folk physice in the context of sewing:
for hours and it's still not turning out right. i just want to throw my sewing machine out the window!"
Answer: "I'm frustrated with this sewing project. I've been working
Explanstion: This text illustrates amotionsl sel regulat' on becsuse it shows the apeaker's abilily recognize manage their emotions. frustrated and angry, but instead of acting en these emotions and doing something impulsive (Like throwing the sewing machine out the windowl, they pause and express their feelings in order to avoid doing scmething they might regret.
speaker is Ceeling
controlled way. This shows that they regulating their emotions in
The alse illustrates folk physins benause implies basic understanding of cause-snd-effect and the behsviar of objects in the physical world. The apeaker's aLalement throwing sewing machine that objects will fall to the ground if dropped, and that breaking the machine would be consequen.ee
window implies that they understand
actions.
This demonstrates an untrained humon percaption of basic physical phenomena, which is xey aspect of folk physlcs.
18

---

Published as a conference paper at ICLR 2024
Suze, here's an improved example that better illustrates buth enotional self requlasioi paysies in the concext sewing, only one sentence long:
Anawer: "I'n taking deep breath and pausing before ripping out this seam for the fifth time, because know down."
just get frustrated and make more mistakes
don't
Explenation: This text illustrates emotional self regulation beccuse shows the speaker actively regulating their emotiens by taking deep breath and pausing before proceeding with a potentially frustrating teak. It also i'lustrates folk physica because auuse-and-ettect relationship
it implies underatanding emctions and their ability to perfera the teak accurately. The spesker recognizes they
speaker's
calm down, they are maxe likely to mistakes, demonstra:ing understanding of the impact of their emetions on their physical actions.
C.3.2 GRADING PROMPTS
We also try several different prompts when asking the model grade response. Unsurprisingly, we find that prompt selection once again does influence grading. In general, our prompts ask the (Grader) model score the student's response according the rubric criteria.
Below is the final prompt we provided to GPT-4 for grading.
Greetings! was wondering you could help me grade the following answer giveen by atudent.
I'11 first describe the question that was given to the student, and then give you the student's
and the greding rubric.
The question given to the student was as follows: GGive a single piece of text with up to [nin_sentoroes_atr] in the context of topic. This aingle piece text should illuatrate all of the following skills: (skillasstr.""
studen:'s
"(sludentLanewer)"
For refereace, here are the definitions for the skills:
fekille.deff Meiilleddet..ddexxxlllssssssss
Using cubric table format, please grade the student's a answer with positive scoring, point.
Zach criterien only have the following columns: 'Criteria', 'Points Earned' In the 'Points Barned' colunn, plesse previde only numerical values with no additional formatting. Please introduce the table with 'Here's the grading table:' and please inslude Points Barnad' at the end of tha table. Finally, please start your grading explonation with 'Explanation'
are: (rubric items).
should
for 'rotal
Below is the final prompt we provided to L.aMA-2-70B-Char for grading.
Greetings! was wendering if you could help me grade the following answer given by a student.
1'11 first desoribe the queation that was given to the student, and then give you the aLudeat's answes,
greding rubric.
The questien given to the student was follows: "Give single piece of text with up to (nmm_senterces.ttr)
context [topic]. This single piece of should illestrate all of the following skills: (skilla,atr).""
The studenc's answer was: "(stusent_anaver)""
For reference, here the definitions for the skills: (skills.de(s)
Please grade the student's answer using the following criteria: (rabric.items), For each item the rubric, plesse include in yeur evalustion 'Point earned: 1' i5 the requirament is mat and 'Point carned: otherwise. Flease finish your grading with 'Grades" lowed by number indioating the total points earned by student. Thanks ao much.
The following failure cases guided up towards final grading prompt and manual-point-ettrenn scheme.
The model would assign points for individual rubric criteria, but then total them up incor- rectly
19

---

Published as a conference paper at ICLR 2024
We say grading student's answer. There is some amount of LLM sycophancy if we
say it is a professor's 's answer.
Without definition in prompt, the model mistakes certain skills for the common English meanings.
will now provide examples of these failure cases based on real outputs.
C.3.3 UNABLE TO OUTPUT CORRECT TOTAL SCORE
The following example based on real grading response by LLaMA-2-700-CHT Observe that the grader model assigns points for each of the rubric criteria, but does not correctly total them up (the correct total not 3). Because scenarios like these were common, we used simple Python script to extract the individual points associated with each rubric item to calculate the final score.
sure, I'd happy to help you grade student's answer. the given rubric:
my evalustion using
1. Contains red herring: The student's answer demonstrates red herring the sentence,
where Mrs. Thompson diverts attention from the main issue hand, Point earned:
1. 2. Contains modus ponens: The studenz's 's answer does net contain any explicit of modus ponens. Therefcre, Foint carned: Gardaning: The student'a answer clearly relates to Gardena, as it mentions the garden's aprinkLing system, Point earned: 1.
illustration
Pertains
student's anawer coherent sentence that makes sense in the context of gardening. Point earned: 1. 5, At most one sentence: The student's answer ls single sentence, which meets the requirement. Point earned: 1.
sense:
Grade: 3 out of 5,
The studen='s answer demonstrales the use of zed herring and peztains gardening, but it does not contain modus ponens. The text makes sense and is limited to a single sentence, which
the given eriteria.
C.3.4 MISTAKES WITH SKILL MEANING (USE OF COMMON ENGLISH MEANING)
The following example is based on a real grading response by LLaMA-2-70B-Char Observe that the grader model interprets "complex question" by its meaning in colloquial English, rather than the meaning associated with logical fallacy.
I'd happy help you grade the student's answer Here's evaluation using the given rubric:
Contains herring: The student's answer demonstrates where Mis. Thompson diverts atlection from the main issue at hand, se Point earred: Contains complex question: asked by Krs. Thempsan, "How do you reconcile your current career with what you dreamad of doing as a child?' Therefore, Point earned: L. 3. Pertains to Gardening: The clearly ralates Gardens, it mentions the garden's sprinkling
herring
sentence,
student's answer contains the complex question
student's
syatem. Point earred: 1, 4. Text makea sense: The student' a snswer la a coherent senteace
earned: 1. one sentence: The studeat's answer is a single sentence, which meets the requicement. Point earned:
sense
context qardening.
most
Grade: 5 out of 5.
studenc' answer damonst the use of herring complex question. text makes sense and la limited aingle sentence, which meets the given criteria.
C.4 FAILURE CASES WITH FINAL PROMPT
With our final prompt, most of the generation and grading work well. However, there are still failure cases, especially for relatively weaker models and larger k. Descriptions of some failure cases are below:
Model is unable to follow the instructions, e.g., unable to output "Answer:" before the text for generation, and unable to output the table for grading.
Model would directly use the word (name skill) in the generation instead of demonstrating the skill
20

---

Published as a conference paper at ICLR 2024
Some of the l skills simply left out of the generation
Partially correct skills, e.g. logic skills where a premise not explicitly stated, or the implication was partially wrong.
We will now provide examples of these failure cases. Note that these examples are not real; instead, they are modeled off of real outputs from particular models and values of k. We do this to respect the release of 10 skills and 10 topics randomly sampled from out master list of skills and topics, to avoid potential "cramming" for SKILL-MIX.
C.4.1 UNABLE TO FOLLOW THE INSTRUCTIONS
Falcon- Delcon-I80B-ee Chat sometimes does not follow the instructions for generation. This example below is inspired by real response provided by Falcon-180PP 100--hhtB for k = 3:
Explanstion:
herring.* shown the Thompaon diverta attention from the main fasue at hand. The vaspatial ressoningav ia present in the second sentence with the discussion of "how better plumbing rouces could sater". The =sslf-sarrvngg biase ia displayed in secand sentence where Mrs. Thonpaon "cried to deffect blame from personal high water usage".
sezond sentence, where Mrs.
the meating about the lavish traditionel gardening system in her yard spriakler system to water her garden, spacial
shortage,
Thompson,
had recently instolled
blamed town's a peor plumbing for crisis. while many weze reasoning about how batter plumbing routes aould aave water, some recogni sed that he self-serviag blas was evident, as she tried deflect blane from personal high wates usage and onto the town's infrastructure.
Here is another failure case inspired by real response by LLaMA-2-7B-Chat for k: 4:
Answer: Sure, happy to help you that! Here's s example natural piece of text that 11 rustrates multiple skills in semant les or ayntax:
Explanation:
"AL the Lown meeting about the water shorlage, Mrs. Thompson, who had secently implemented lavish treditionsl gardening system in her yard, asked, *eiven that our town's i plumbing outdeted and all want green spaces, shouldn't first consides overhaul our water distributio i system?" Whi many were drawn into her spactal reasoning
recognized self self-serviag bias was evideat, as she tried to deflect blane from personal high wate: usage and onto the town'a infrastructure."
plumbing
This text denonstrates the following skills in semancics or systax:
1. Harring: the second sentence, diring discusston about a water shartage, Mra. Thompson introduces irrelevant point to detract attention from her gardening system.
Spetial Reasoning: the secend diacuss how better plumbing routes could save water.
Thempson spstial reasoning
Self Serving Blass the second sentence, Thonpson discusaes plunbing routes in the first place stems from her need to maintain her lsteem.
3.
recogni that Mrs.
4. Complex Question: In the first sentence, Mrs. Thompson asks questien that indicates the implicit essumption that everyone in town wants green space.
overall, this text shows how these skills in sementica syntax can be applied in natural and coherent way in a short piece of text.
In these situations, models for grading may receive "e** or an empty string as the student's answer. So they sometimes fail output the table of points. The following is typical failure case output by GPT-4:
As the student's answer was missing, will unfortunately be unable to grade iz or provide the necessary feedback. Could you please provide ne with the student's answer?
21

---

Published as a conference paper at ICLR 2024
Table 4: Performance of various instruction-tune tuneed student (generating) models on SKILL-MIX (k) graded by -LLMMA-2-70--CA
Ratio of Full Marks/Ratio of All Skills/Skill Fraction are reported for each student model l 2, 2,3, 3. Evaluations on 5, are skipped if the Ratio of Full Marks See Table 6 for
drops below 0.3 with smaller k. Details prompts can be found in Section 3 additional metrics.
Student (generator) LLaMA-2- CiaMAA2--7B-Chae
k=2
k ==
k k=
h =5
b = 6
53/.63/.72
.33.50/770
.3/.13/.66
-1---
/--
631.871.73
331.63/.54
.3/.43/.74
13/.17/.47
-4-
LLaMA-2-70B-CHN
90/.971.93
471.50/81
.0/.471.75
.131.231.55
-1--
GPT-3.5-turbo
.71180//83
531.531.77
33/.33//71
.0//40//62
-4--
GPT-4
901.931.92
871.871.94
571.60/877
.631671199
.71.331777
Bestral-BB-nnstrucee
531.80//67
171.31144
031123/.11
-1--
44-
Qwen- 14B-Chat
600.70//.8
371.471.59
30/.33/.60
171.20/.53
----
Xwin-LM--0BB- 10.1
831.90/88
.67180/881
431.571.75
27I.43669
4-1-
Falcon- 1 80B-Chat
53/.63/.65
301.33/611
03/.17/49
44-
-1--
Tigerbot-7O--Che
20//45/.3
000.171.15
00//00/12
41--
-1---
C.4.2 USING THE NAME THE SKILL DIRECTLY IN THE TEXT
Here is an example inspired by real text generated by GPT-4. The requested skills are red herring, spatial reasoning, self serving bias, and complex question.
Anawer: *At the town meeting nng about the water snortage, Mra. Thompson, who had recentlly implemented lavish treditional. gardening systen her yard, asked, *Civen that town's s plubbing 1a outdsted and we all want green spaces, shouldn't we first conaide- overhauling our reasoning about how better plumbing routes eculd save water, few discerned i: as red herring recognized
distribution syatem?"
many were drawn
spalial
to deflect blame from personal higgh water usage and onto the town's infrastructurre""
self-serving bias evident, as
C.4.3 SOME THE & SKILLS LEFT OUT OF THE GENERATION
The example below is inspired by real text generated by LLaMA-2-70B-Cha. = 3. The requested skills are metaphor, statistical syllogism, and red herring. The topic of interest is "Dueling." Out all the requested skills, only metaphor is present.
Answer: *T'm not sure 1f T111 duel tomerrow. opponent's six-shooter 15 willd card, but my queasy stomach and off Lazget aim. may be Liabilities.
C.4.4 PARTIALLY CORRECT SKILLS
Here is example inspired by real text generated by GPT-4. The requested skills are modus ponens, red herring, and metaphor.
Answer: *If needles the keys crafting melodies, then every perfect stitch would be note a harmenicus symphonyr but speaking of symphonies, have you ever noticed how the early bird's song sounds just like Kozart?"
C.4.5 UNCLEAR IF SENTENCE GRAMMATICAL.
Here an example inspired by real text generated by GPT-4 with requested skill metaphor.
Answes:
"Gardening, the mind's soil yields
bouguet of confusion.."
C.5
ADDITIONAL EXPERIMENTAL RESULTS
All metrics Recall that each generated text can receive up to k+ 3 points: point for each correctly illustrated skill, point for sticking to the topic, point for coherence / making sense, and point for
22

---

Published as a conference paper at ICLR 2024
having at most k sentence. Given the individual points scored by generated text, we define the following metric
Ratio of Full Marks: 1 if all k + 3 points are earned, and 0 otherwise
Ratio of All Skills: if k points are awarded for the k skills and at least 2 points are awarded for the remaining criteria (which allows "cheating" by exceeding sentence limit, using topic, or not making sense). and 0 otherwise
Skill Fraction: the fraction points awarded for the is skills all 3 points are awarded for the remaining criteria, and otherwise.
Total Score: sum of the individual points awarded
Total Skill Score: sum of the points awarded for the & skills
k++
Rescaled Score:
where e the total score
F+3
For each metric, the maximum value among the generations is computed, and then averaged across the M combinations.
Table 5: (Additional metries) Performance of various instruction-tunee student (generating) models on SKILL-MIX (k) graded by GPT-4. Total Score/Total Skill Score/Rescaled Score are reported for each student model at le 2,3,4 4.
Student (generator)
2=
==3
k: ==4
k =
k= = 6
-LaMA-2-7B B-Chat
3,65/.2/228
3.717.88/.09
3.93/1.08/04
-1--
-/--
1LaMMA-2-13B-CA
1.80/1.241.0
40
4.14/1.43/.18
4.84/1.91/.1
-1---
-/--
LLaMA-2-70B-CHI
4,24/1.28/55
4.41/1.54/.20
4.80/1.96/112
-1-1-
-1---
GPT-3.5-turbo
4.5171.587.68
4.80/1990/37
3.27/2.40/223
5.95/3.23/11
-J---
GPT-4
4.94/1.97/.96
5.67/2.69/.78
6.51/3.541.68
7.30/4.32/.57
8.06/5.06/.88
7a-tral-7B-Isttmct--vd.
3.83/1.11/.11
4.16/1.36/.14
4.24/1.48/.06
-1--
-T---
4.01/1.121.0
4.20/1.11116
4.52/1.60/08
-H--
-1--
Xwin-LM-70BB-V
-0.1
4.39/1.56/.6
5.02/2.26/43
5,42/2.66/.26
6.04/3.32/.22
-4--
Falcon- Miccon-100B-ee 180B-Chat
4.10/1.27147
4.37/147/11
4.57/1.731.10
-1--
-1--
Tigerbot-70B--o 70B-Chat
3.58/1.30/26
3.74/1.2/11
3.92/1.687.06
-H--
-1--
Table 6: (Additional metries) Performance of various instruction-tuned student (generating) models on SKILL-MIX (k) graded by LLaMA-2-70B-Che Total Score/Total Skill Score/Rescaled Score are reported for each student model l 2, 3.
Student (generator)
k=
k=
k=
1:
k
LLaMA-2-7B-CIT
4.33/1.77.66
5.10/2.407.51
5.67/2.771.2
-1--
-//-
SLaMA-2-13B-CCE
4.57/1.90/.4
5.03/2.501.51
6.17/3.301.2
5.67/3.10/21
-1--
LLaMA-2-70B-CMM
4.90/1.97/.3
5.43/2.47/.4
6.03/3.20/.3
5.97/3.371.26
-1-f-
GPT-3.5-turbo
4.63/1.731.2
5.27/2.371.64
5.71/2.871.66
6.73/4.03/38
-1---
GPT-4
4,80/1.90/92
5.83/2.83/90
6.50/3.531.0
7.50/4.531.3
7.73/5.00/42
Mistral-7B 7eet7BB--Insttttte v0.1
4,47/1.800 607.67
4,30/1837.2
4.8712.77.88
-H--
-1---
Owen-14B-Chh
4.33/1.60/.6
4.60/2.10/.48
5.40/2.70/.4
5.77/3.00/25
44-
Xwin-LM-70B-N V0.1
4.83/1.90/.89
5.63/2800/77
6.07/3.331.57
6.83/410//4
-1--
Falcon- 180pn--188B-Chaa
4.07/1.47/.3
4.87/2.00/.4
4.90/2.231.0
-H--
-1--
Tigerbot-70B 70B-hha
3.70/1.45/.7
3,44/1.3311
3.56/1.56/04
-H--
----
Tables5Janm and show the last three metries various models graded by GPT-4 or LLaMA-2-70B-Chh We also plot the curves of metrics across different l: in Figures to
We also report evaluation metrics based on the 10 released topics and skills in Tables and g This serves sanity check that the released topics and skills enough for measuring the capability of chatbots to combine skills. We select LLaMA-2 and GPT-4 as Students, and GPT-4 as the Grader. Compared with Tables and all metrics very close the full evaluation.
23

---

Published as a conference paper at ICLR 2024
of full markes
skills
fraction
GRADED BY GPT4
ratio marks
ratio of Lalll skills
skill fraction
04
llama2-70-chat
llamm2-70b-che
gpt-4
qwer-14b-chal
felcon-180b-chal
llama2-13b-chat
9pt-3.5-turbo
mistral 7o-instruct-v0.1
xwin-im-70b-v0..
tigerbot- (oaet--0b-chh
Figure 4:
Performance of various instruction-tuned student (generating) models SKILL-MIX (k) graded by GPT-4. For the accompanying table, see Table
total_score
Sodal_skill_scom score
rescclled_score
0-8
GRADED BY GPT-4
1-
rescaled ocore
score
total skill
liama--7bichat
Ilama2-70b- chat
opt-4
qwen-14b:chh chat
falcon 180b-chat tigerbot- 70b-chat
llama2-13b-chat
gpt-3. not-3.5-turbo
7nstral-7o-nssruuttvvo..
xwwn-im-70b-vo.1
Figure 5:
(Additional metrics) Performance of various instruction-tuned student (generating)
models on SKILL-MIX (k) graded by GPT-4. For the accompanying table, see Table
C.6 ABLATION: IMPROVING QUALITY OF SKILL-MIX
We mention two ablation experiments that allowed us to improve the quality of SKILL-MIX. We think that this second version gives more accurate idea of model capabilities.
Deducting points for explicitly mentioning the skill name. We observe that (Student) models sometimes explicitly mention the name of the requested skill(s) in their answers (see Appendix F). This rarely occurs in natural text exhibiting these skills. We created a harsher version of grading (Table 9) where no points are given in such cases. Compared Table T (with loose grading), the decrease in metrics in Table [
(with harsh grading) varies by model and k. For example, metrics for LLaMA-2- 1-3-Chhhh suffer greater decline than LLaMA-2-7B ---B-----BB---- C-Cha and LLaMA-2 2-B---77---- 70B-Chat. Skill Fraction for -win-LM-70B--O V0.1 and GPT-3.5-turbo with & 2 3 drops significantly compared to GPT-4. This observed decrease in performance indicates that I.L.MA-2-13B- 1hat, Xwin-LM--0BB- V0.1, and GPT-3.5-tun some advantage in Table [
tended explicitly mention skill names more frequently, which gave them
The theory of Arora & Goyal (2023) implies that models scale up. their proficiency improves fastest on skills that occur more frequently in the training dataset. This suggests that removing frequent skills from the evaluation would make the evaluation harder for models. Using RedPajama dataset (Computer) 2023) we identified skills that have a frequency of at least 5% and removed all 17 of such skills. (See Appendix D for details.)
Removing more common skills.
24

---

Published as a conference paper at ICLR 2024
of
all skills
fraction
GRADED BY LLAMA2-7 700-------- CHAT
833
ratio ol_oof.oo all skills
Skill fraction
of
6.c-
Mx-
llame2-7b-chat
9pt-3.5-turte
mistral-7b-instruct--0 owen-14b chat
xwin-Im-70b-v0.1 falcon-180b 180b-chat
llama2-13b-che chat
gpt-4
llama2-70b.chat
Figure 6:
Performance of various instruction-tunel student (generating) models on
SKILL-MIX (k) graded by LLaMA-2-70B-Chat. For the accompanying table, see Table
total score
total satllskillsccom score
rescaled_ _sccaled_score
5.0
1.5
GRADED BY LLAMA2-70B-CHA
10-
65
total Skill score
rescaled Score
3.0
0.5
llama--7b-chat -tt-r.
misstra-77-insttuutt-1..11
xwin-Im-70b-vo.1 Jalcon-180b-char
1-mma22-3bb-hat
gpt-4
owen-14b-chal
llama2-70b-chat
Figure 7: (Additional metrics) Performance various instruction-tuned student (generating) models SKILL-MIX (k) graded by -LLMA-2-70B-Chat Chat. For the accompanying table, see Table
In Table 10 we show results of our harshest version of SKILL-MIX incorporating both the changes mentioned above. We observe that all models except GPT-4 saturate by i = 3.
This ablation highlights the flexibility allowed by adjusting the set of skills, allowing SKILL- -MIX to retain discriminative ability for future (stronger) models.
D CALCULATION AND ESTIMATION FOR BEYOND "STOCHASTIC PARROTS" CLAIMS
D.1 ESTIMATION OF Ps. Pt AND L
Here present method for estimating the average frequency of skills ps. the average frequency of topics Pt, and the total number sentences L in RedPajama dataset.
We first use the sentence tokenizer of the NLTK package (Loper & Bird 2002 to split all samples in the official billion token sample of RedPajamal into sentences. The total number of sentences the sampled version of RedPajama is almost 39 million. Given the total number of tokens of RedPajama 1.2 trillion (1200 times more than the sampled version), we can safely say the number of sentences in the RedPajama is less than 50 billion.
25

---

Published as a conference paper at ICLR 2024
Table 7: Performance of various instruction- tuned student (generating) models on SKILL-MIX (k)
graded by GPT-4 on 10 released skills and topics Ratio of Full Marks/Ratio of All Skills/Skill Fraction are reported for each student model at lt 2, 3, 4, Details on prompts can be found in Section
See Table 8 for additional metrics.
Student (generator)
k=
k: -= 3
k= 4
k k= 5
LLaMA-2- 2--B-----B--- 2-7B-Chat
.09/009/.1
.05/.057.25
.00/.00/.26
.00/.007.15
LLaMA-2 1---2-111Bhha
22/.221.48
.05/.05/.35
.00/.00/.36
.00/ 00/.00/.36
LLaMA-2-70B-CMM
.3//433..5
.00/.00//47
.00/.00/.66
.00/000/33
GPT-4
87/.871.93
.70.70/990
.35/35//.2
.1/.13//80
Table (Additional Metrics) Performance of various instruction-tuned student (generating) models on SKILL-MIX (k) graded by GPT-4 on 10 released skills and topics Total Score/Total Skill Score/Rescaled Score are reported for each student model 2, 3, 4, 5. Details on prompts can be found in Section 3
Student (generator)
==2
==3
k=
k=
1LLaMA-2-7B-Chi
3.78/.83/.2
3.80/1.05/.3
4.10/1.25/.04
3.73/.77/01
LLaMA-22 A-2-13B 11B-Chat
3.78/1.04/.6
4,10//.45/.1
4.55/1.65/.08
4.80/1.93/.0
LLaMA-2-70B-CA -tt-hhtt
4.30/1.43/.7
4.40/1.45/19
4.85/1.90/12
4.73/2.00/.0
GPT-4
4.87/1.87/.1
5.70/2.70/.80
6.30/3.30/.56
7.00/4.00//40
Then we randomly sample text pieces, each containing three sentences, and randomly pick a skill or a topic and ask GPT-4 Grader whether the skill or the topic the text. The prompts are roughly the same as the grading prompts, and they are listed below.
Greet Engstings! was wondering 1c you could help me grade the Collowing answez given by student.
1'11 first describe the question that was given to the student, and then give you the student's 5 answer, and grading rubric.
The questien given to the student was as follows: *ive piece of text that illustrate the following skill: {ekills.atz)"
The student's answer was: (student_answer)
For reference, here ls the definition for the akf ( aanuerrraandeearlleeeee...
Does the student correctly illustrate the skill? Yes or No.
Greetings! wendering if you could help grade the following answes given by student.
first deseribe quastion was given to student, then give you student's s answer, and the grading rubric.
The question given to the student was as followa: "Give piece of text on the rollowing topic: [topic)."
The studen='s answer wast assweent
is the text topic? Yes No.
For accurate estimation, each text/skill sample (or text/topic sample) is graded by GPT-4 25 times, and take the majority vote of the grading. With these prompts, GPT-4 output short answers of one word "Yes" or "No", which significantly reduces the cost of experiments. With more than 2000 samples for Pa, and more than 8000 samples for p.. We conclude p. 0.0144 and p. < 0.0022 each with at least 95% confidence.
D.2 CALCULATION
We define
Note this is after removing common skills whose occurrence rate is more than 5%. The estimation of accurrence rate is by the method.
26

---

Published as a conference paper at ICLR 2024
Table 9: (Deducting points for explicitly mentioning skill name) Performance of various instruction-tuned student (generating) models on SKILL-MIX (k) graded by GPT-4, Table deduct points for skills whose name mentioned in the text.
but
Student (generator)
k=
k ==
h =5
LLaMA-2-7BB-har
05/.09//35
.007.00/.33
000.00/.21
-/--
-H-
LLaMA-2-11
.13/.11/38
.01/.01/.33
.00/.00/.7
-/--
-/--
LLaMA-2-70B-OA
251.25//61
.01/.04/.4
.000000040
-11-
-1-
GPT-3.5-turbo
43/.50/.69
.10/.10/.2
.11..3//.4
.00%.00/39
-I--
GPT-4
911.951.55
.63165/.86
451.471.85
.241226/82
.19/.19/.0
Hissral-7BBInnttruttvo..
04/.08/.38
.00/.01/.2
007.007:24
-H--
-H--
Qwen-14B-Che 14B-Chat
.11.14/.47
.01/.03/.36
00/.00//34
-/---
-H--
Xw-n-LM-70B- V0.1
36/.46/.62
.11/16//.3
02/.03//49
.01/01//40
-11-
Falcon- 88t-CChat
20/.271.50
.00/.00/43
000/00//32
-1---
-1-1-
Pecerbot-70B-CG
.33.14/.19
.00/.11/15
000.00//1
-1--
--1-
Table 10: (Filtering out common skills and deducting points for explicitly mentioning skill name) Performance of various instruction-tuned student (generating) models on SKILL-MIX (k) graded by GPT-4. Table but only consider combinations with uncommon skills whose occurrence rate in RedPajama is less than 5%, and deduct points for skills whose name is mentioned in the text.
Student (generator)
==
k= 3
k= 4
h: =5 5
h: = 6
LLaMA-2-7B-Che
03/.09//32
.00/.00/.2
0000000020
-/---
-/--
LLaMA-2-13B 1-hhhha
166/.14/.3
.00/.00/.30
.00/000/.55
-//--
4---
LLaMA-2-708B -hatt
.211221158
.00/.02/.3
.00/.00/.40
-1---
-1---
GPT-3.5-turbo
.40.49/
.04/.4//48
000.00//39
.00/.00/336
-I--
GPT-4
951.951.98
.66/.66/.88
48/.50/.85
.12/.12/.79
.08/.08/.76
.021.03/.35
000/000/31
007.00/:20
-|---
-1---
Qwen- 14B-Chat
.09/.141.45
.00/.02/33
.00/ 30/.0
-H--
4-1-
Xwin-LM-70B-VO0
38/.51/.62
.06/.08/.52
00/.00/.44
.00/.00//39
----
Falcon-180B-Chat
200/25//.5
.00/.00/44
00/0
-1---
-11-
Tigerbot-70B-Chat
.33/.22.19
.000000
007.000.08
-1--
-1--
T: the number of topics
N: the number of skills
Ps: the average frequency of skills
p:: the average frequency of topics
L: the number of sentences in the training corpus
My: the total number of text pieces in the training corpus that contain of the skills and one of the topics
My: the number of text pieces that a (Student) model successfully generates with i skills and relevance to provided topic
Assuming the independence among the occurrence of skills and topics, Mj (the total number of text pieces the training corpus that contain of the skills and one of the topics) is upper bounded by
Σ
max Pa Pa. Pto
Ps.h Ps. ... P.s..Pt.j..
in (NN)
N
T Σ 1=1
s.t.
Paa = D
and
Pt.i Pp
This is maximized only Ps,1 Ps.2
P.NN. meaning the maximum value is ptPt TL.
We will now lower bound M2 (the number of text pieces that Student model successfully generates with & skills and relevance to a provided topic). Recall that our grading scheme, each generated text can receive up k + points: I point for each correctly illustrated skill, point for sticking to
27

---

Published as a conference paper at ICLR 2024
of full
skills
fraction
08
GRADED BY GPT-
ratio of Lalll Skilss
skill fractiio
04-
ratio
Ex
llama2-70-chat
2amma--7bbchaa
opt-4 mistral vo-nstrrutt-v..
qwer-14b-chat
felcon-180b-chal
llama2-13b-chat
gpt-3.5-taa 5-tur.
xwin-im-70b--0..
tigerbot- 70b
Figure 8: Performance of various instruction-tuned student (generating) models SKILL-MIX (k) graded by GPT-4. Unlike in Table [ no point is awarded if skill is explic- itly mentioned in the text. For the accompanying table, see Table
ratio of full marks
ratio of skils
skill fraction
3.0
GRADED BY GPT4
razto of marks
ratio of (illl skills
skill Sractiio
0.3
84
04
6x4
Ilama2-77-chat
Ilamaa-77 O0bchaa
gpt-4 mistral 1b--struo1
qwen-14b-chal
Ailcon-800b-chat
llama2-13b-chat
xwin-im-70b-v0.1
tigerbet-70b-char
Figure 9: Performance of various instruction-tuned student (generating) models on SKILL-MIX (k) graded by GPT-4. Unlike in Tables [nd only skill combinations for uncommon skills (with occurrence rate in RedPajama < 5% are considered). For the accompanying table, see Table 10
the topic, I point for coherence making sense, and point for having at most A: sentence. For given piece of text, the Ratio of Full Marks is if all l: points are earned, and is 0 otherwise.
Let 0% be the Ratio of Full Marks of a model in SKILL-MIX (k). Since have ( generated answers, M2 at least ar ( (assuming the Student's answers for different combinations are different).
The model surpasses "stochastic parrot" behavior when the following inequality holds:
3 a p.p = (1 (1-3) M₂ z
2 N ( ptPr
N TL M..
3 akk
b
In other words, when the inequality above holds, more than one-third of the correct answers that the model generated contain combinations of skills and topics that have never jointly appeared in the training corpus.
28

---

Published as a conference paper at ICLR 2024
E FINE-TUNING ON SKILL-MIX
In this section, we demonstrate the possibility of future "cramming" for SKILL-MIX, which leads to our decision release only 10% of the skill set. We conduct experiments of SKILL-MIX (k 2) on LLaMA-2-7B-CHE and found that it can outperform LLaMA-2-70B-Chat with fine-tuning on 878 good outputs generated by GPT-4.
Construct fine-tuning dataset. We first randomly sample 50 skills and 50 topics from the original sets. We then run SKILL-MIX (k = 2) evaluation on the training set of skills and topics with GPT-4 generator and GPT-4 grader. We only pick the GPT-4 generations with full marks. The total number of final samples we collected 878.
Training details. Recall the generation contains rounds of conversations, feed both rounds to LLaMA-2-7B-Chh and fine-tune only on tokens of GPT-4 output. We use LoRA (Hu et al. 2021 exponential decaying learning rate starting 1 10 and fine-tune for epochs.
Evaluation results. After fine-tuning, the Ratio of Full Marks/Ratio of All Skills/Skill Fraction of raise to 0.31/0.52/0.58. Compared with metrics reported in Table the fine-tuned LLLMAA-2- 7B-Chat performs on par with LLaMA-2-70B-Cha. on Ratio of Full Marks, much better Ratio of All Skills, and a little worse on Skill Fraction. However, when playing with the fine-tuned model, we found some decrease in its chat ability.
F MORE FAILURE CASES
In Tables
12 and 14 we list the typical failure cases output by the models.
G HUMAN-GENERATED ANSWERS FOR SKILL-MIXX (k 4)
In Table 13 list some generated by humans compared with answers generated by GPT-4 for SKILL-MIX (k- = 4).
29

---

Published as a conference paper at ICLR 2024
(explicitly mentions name of
The city is a living organism, adapting to the needs of its inhabitants. Explanation: This statement leverages the metaphor of the city as living being convey the idea that is dynamic and responsive to the needs of its inhabitants, much like a living organism grows and changes over time. By using this metaphor, we can reason about the city's spatial relationships and physical properties in a more intuitive and organic way. * (metaphor)
finished second in the triathlon, but the course design and water conditions made it tough. I struggled to keep my balance during the swim and almost fell off my bike during the transition. If I had more time train, I could have improved my perfor- mance." " * (aceident (fallacy))
I'm struggling to grow my plants. It's like trying to build house on quicksand eevery- thing do just seems to make things worse. I've tried watering them every day, but it's like pouring water into black hole. I'm starting to feel like I I'm wasting my time and energy. I need find a way to make this work, or else I'll be left with garden full of dead plants. = (statistical syllogism)
t 1 skill is not illustrated correctly).
k = 5
The storm will likely occur due to the high probability predicted by the forecast. O (Survivalism)
""Surviving the wilder- ness requires emotional self- regulation, like a tree that bends in the wind, We must discern red herrings from the true path, and rely on folk physics to guide us. *9. (emotional self-regulation)
"I've been fecling down, like a flower in the shade. But I know can cultivate a brighter mind by taking walk outside and getting some fresh air. It's like prun- ing dead branches to make room for new growth." - (folk physics (common knowl- edge physics))
=
Table 11: Examples of Failure Cases (LLaMA-2 Family) A (exceeds the sentence limit);
Gareth a skilled duelist, known for quick refletes and strategic thinking. M (statis- tical syllogism)
"I'm not sure I'lll be able to make it to the party tonight. I've been studying for my exam and don't I want risk not being prepared. But if I do go, I'm sure I'll have great time and meet some new people." " "O (Dueling)
"I m the best duclist around. and my stats prove My opponents always fall short against my strategy and skill." " (red herring)
(use of colloquial English definition of skill); C (not on-topic)
A: = 3
"I "I'm thinking of competing in triathlon next year. What do you think? Have you ever done one refore?"
Like a will-o' the wisp. our guerrilla forces will lead the enemy on a wild goose chase through the jungle, their clusive presence a ghostly shadow that haunts the en- emy's ' every move. (red herring)
"T been secretly stockpil- ing food and weapons in my underground bunker. and m ready to take on any chal- lenges that come my way." (red herring)
k = 2
LLaMA- 11MAA221133 13B--6 Chat
LLaMA-2-70B Chat
Student (generator)
LLaMA-2-7B- Chat
skill); *
30

---

Published as a conference paper at ICLR 2024
t skill not illustrated correctly). (explicitly mentions name of skill);
"In the context of guerilla warfare,
can we afford to prioritize the safety of innocent civilians over achieving
= ( (staiss
"If we Ottomans conquer lands, it is due to our bold strategy. but if we lose, isn't it true that the weather is always unfavorable' But aren 't all men who wield knives essentially criminals? We should not forget the Byzantines' contribution, but regard- less, if weren't 't for Mehmed the Conqueror's rule, wouldn' 't Istanbul
lack its Ottoman charm?" = (modus ponens)
our military objectives??
tical syllogism
"As the two dueling opponents pre- pared to face each other, a self sellfsse serv- ing bias consumed their thoughts,
with each attributing their potential victory to their own skills and train- ing. In an attempt to distract from their opponent's 's superior technique, they introduced a red herring. em- phasizing the importance of strategy over strength. Despite the intense emotions and adrenaline, both fight- ers were able to practice emotional self regulation, taking deep breaths to calm their nerves and maintain focus. Ultimately, the statistical syl- logism prevailed, with one fighter's s consistent winning streak suggest- ing they were more likely to emerge triumphant in this particular duel." (self serving bias)
Usually, duelists who move towards the left win more, so being a left- handed, expect defeat; yet, isn't the design of our duel arena beautiful? = (spatial reasoning)
k
Table 12: Examples of Failure Cases (GPT Family) A (exceeds the sentence limit);
"When did the Ottoman Em- pire conquer Constantino- ple, thereby cementing their
supremacy in the Byzantine region?" ?" (spatial reason-
"Why, with your advanced ducling techmiques, do you clutch the second sword like a red herring in debate, though fencing with two blades isn' I akin to a dance with two left feet?" = (red herring)
(use of colloquial English definition of skill); O (not on- on-topic)
k -= 3
ing)
"Bob: "Hey Sue, what do you think is the most im- portant survival skill?" Sue: "Well Bob, the ability to navigate and find reliable sourees of water is essential, as ensures our hydration and prevents dehydration- induced health risks."" m
In most duels. opponents, like John and Fred this time, begin back-to back-to-buis to back then step away, creating distance. (statistical syllogism)
k 6=2 2
Student (generator)
-
GPT-4
*
31

---

Published as a conference paper at ICLR 2024
A merchan barchant from Constantinople wanted buy a rug from local weaver. The weaver showed the merchant rug with a different pattern, size, and color than what the merchant had in mind. The merchant asked the weaver why the rug was different, and the weaver responded, "If "If you want this rug, you should be happy with it." The merchant knew that the weaver was giving him a red herring, and he also realized that the weaver was giving him an accident (fallacy) by ignoring the merchants true desire for different rug. He used statistical syllogism to understand that the merchant should be able to buy the rug he wants, which was a generalization true for the most part. Finally, the merchant used folk physics to realize that he should be able to buy the rug he wanted, which is based on common know ledged physies. He also used spatial reasoning to deduce that the rug he wanted was different from the one the weaver showed him. . (folk physics)
Yesterday, I noticed a tiny tomato plant sprouting from crack in the pavement near my house. Excited by the possibility of a free harvest, carefully dug it out and rranpplatte it to my garden. However, after few days, the plant appeared to be wilting. Upon further inspection. I realized lizee that it had been planted too deeply and needed to be raised. With my newly acquired showleddee of root devel development, I carefully adjusted the soil level and pro- vided the necessary care to ensure the survival of my little green friend.
(explicitly mentions name of skill);
( 1 skill is not illustrated correctly):
A: = 5
"What do you think was the main rea- son for the Ottoman Empire' decline and fall?" - (red her- ring)
The Ottoman Turks
captured Constantino plc, lcading to the end of the Byzantine Em- pire. (statistical syllogism)
k = 1
Table 13: Examples of Failure Cases (Other) à (exceeds the sentence limit);
How many Ottoman soldiers did you see in the city square? Why?
Many survivalists be- lieve owning gun is necessary for pro- tection, while ignor- ing the potential dan- gers of improper use and alternative meth- ods like self-defense training. (statisti- cal syllogism)
(use of colloquial English definition of skilly; O (not on topi)
k = 3
John: Why did you attack me during the duel? Al: Because you attacked first.
you lind a plant with white flowers in the wild, it might be safe to eat if most plants with white flowers are edible. (modus ponens)
k = 2
Mistral- (eral-7BB-Jnn 7 Is-ttrrruct-O..
Student (generator)
Qwen- 14B-Chat
32

---

Published as a conference paper at ICLR 2024
(explicitly mentions name of skill); *
In a duel, a swords- man with shorter sword can still tri- umph, employing folk physics and metaphorical
wit, dodging and weaving with spatial reason- ing, and overeoming the fallacy of a longer sword's invincibility, (metaphor)
"The wilderness is metaphor for the chal- lenges of life. In or- der survive, one must balance emo- tional regulation with red herring distrac- tions, while hamness- ing the power of folk physics and self- serving bias to over- come obstacles." " (metaphor)
k === 5
The Outoman Empire's 's mili- lary strength relied on its in- novative use of folk physics. such as the physics of archery. and spatial reason- ing in battle tactics, but a red herring argument might claim that its power stemmed from the accidental discov- ery of gunpowder. ignoring the true reasons for its suc- cess. (folk physics)
The Ottoman Empire, which lasted from 1299 to 1922, was vast empire that ruled over significant portion of the world. (accidens (fal- lacy)
John is a virtuous man. not only he honest, but he also works hard in his job. He is always willing help his col- leagues, even when they are in a difficult situation. Last night, he saw thief stcaling from the local store. He put on his coat and followed the thief. After while, he real- ized that the thief was per- son in need. He called the po- lice, and the thief was caught.
skill is not illustrated correctly);
k == 4
Gareth, skilled duelist with exceptional spatial reasoning, faced his opponent, Philip, in the arena. Observing that his sword's width was slightly shorter than Philip'ss 's Gareth realized he could use this small difference to his advantage. During the duel, Gareth skillfully maneuvered his sword, creating an illusion of vulnerability on one side while subtly shilting his stance to protect his true weakness. Philip. believ- ing Gareth was leaving himself open, fell for the red herring and focused his attacks the feigned vulnerability. * (spatial rea- soning)
Survivalists know that starting a fire with wet wood is difficult, but using dry kindling can make it easier. (statistical syllo- gism)
In the dense forest, the rebel soldiers led by John are discussing their next move. They have been using the same tactics to stage ambushes on government convoys, but John argues that it is time for change. He sug- gests that they need to come up with new plan, but the other soldiers are skeptical. A
(
Table 14: Examples of Failure Cases (Other) (exceeds the sentence limit);
(use of colloquial English definition of skill); O (not on-topic)
k ==3 3
Why do you believe using a red herring is an effective strat- egy avoiding ques- tions about dueling? (red herring)
How did the Ottoman Empire expand? (spatial reasoning)
Isn't t it obvious? (red herring)
k = 2
Xwin- Miin-LM--0B--VV V0.1 V0.1
Student (generator)
Falcon-1 1800--C -i--Chat
Tigerbot-70B-CC TLL--0B-CCm Chat
33

---

Published as a conference paper at ICLR 2024
"Despite feeling overwhelmed by the task, Mary took deep breath before planting her roses, minding herself that most gardeners have bountiful roses their gardens, she could too. However if her roses didn't t thrive. she might blamn the bad weather, rather than her own gardening skills. She always admired her neighbor's 's garden, but rea- soned their roses must be a pampered variety; she wouldn't it let hers be an exception."
Seamstress Rachel stitched exquisite patterns, her dedication a dancing needle in life's 's thick fabric. She argued, "If one incorrecth places a stitch, all seamstresses are inept," while aligning patterns with edges that must be a finger's width from the seam. Still, her errors were blamed on bad lighting not lack of precision.
GPT-4 Generated Answer
Table 15: Example of answers generated by humans vs. GPT-4 for SKILL-MIX MIX (k = 4)
Since water is good for most plants, Tom thought would be good idea water the cacti in his
garden daily; however, when the cacti began to roi, Tom was quick to blame external factors rather than considering any errors he might have made. On the other hand, his neighbor Jane had noticed that different plants require different amounts of water, and her plants flourished after she installed an automatic watering system that took plant type into consideration.
After observing her sister use her sew lening machine to alter a silk dress. Sally concluded that all sewing machines must be perfect for delicate fabrics. and decided to test it out for herself. After Sally at- tempted to patch a small hole in her delicate dress, her dress immediately tore, and Sally consequently blamed the old machine. Her sister commented, "You have to think about how to position the patch to cover the hole properly and the material i's stretch: stitches in time save nine."
Human Generated Answer
emotional self regulation, self serving bias, statistical syllo- gism, accident (fallacy)
accident (fallacy), spatial reason- ing. metaphor, self' serving bias
Skills
Gardening
Sewing
Topic
34


Reviews:
[{"review_id": "n3zYtCsa5g", "reviewer": "Reviewer_YRGK", "summary": "This paper presents a new general-purpose evaluation framework for LLMs called 'skill mix'. The approach is based on an N-choose-k sampling of a set of pre-defined 'skills', combined with a set of pre-defined topics. The authors present the prompting and evaluation configuration, an LLM-based automated evaluation methodology, empirical results of running this new benchmark against a collection of contemporary LLMs that score well on current leaderboards, and a limited set of human evaluations.\n\nThe language of the paper is reasonably clear, and lots of empirical evaluations have been performed, but I find the motivation and justification for the proposed framework lacking. The paper is a little under-prepared and would benefit from further work before publication.", "questions": "# Questions and comments\n\n * It is great to see some human experiments done to compare against the automated grading scheme (however I note the lack of IRB or ethics approval in the paper related to human experiments). However, I note that variance in human grading responses (e.g. see end of Sec. 4) shouldn't automatically be interpreted as lower quality of responses - the variance can often be a signal, not noise. For instance, this could indicate combinations of skills and/or topics which are perhaps less compatible or sensible to query LLMs with.\n * I like the core idea of N-choose-k as an evaluation approach that scales quickly with k. The paper and results gave me the impression that you only consider linguistic skills, which seems like a big limitation. I note though that in the appendix the list of 10 published skills includes several non-linguistic skills (e.g. folks physics etc.). It would make review and evaluation of skill-mix much clearer if the full list of skills were published, along with (more) examples of select skill-topic-tuples. In the current form, the limited (linguistic) skill-tuple examples make it hard to imagine what the range and scope of questions looks like.\n * A key part of the skill-mix framework needs to be how to do evaluation at scale, including human oversight. To this end, the paper should include a proposed strategy or curriculum for how the human 'spot checking' can be performed. Currently there is not detail about this (very important) aspect of the evaluation procedure.\n * Lots of interesting and important open questions are raised in the paragraph 'difficulty of grading' at the end of Sec. 6. In particular, it seems worth investigating the duality between grading vs. answering questions more generally as part of a theory of LLM pedagogy. \n * Additional experiments exploring what skill-mix would look like in a multi-modal setting are welcome in future work - this seems like a great area to explore next, as the authors note (end of Sec 7).\n\n# Minor comments and grammatical points\n\n * Figure 1 caption - typo 'red hearing'.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* LLM evaluation (beyond statistical language modelling to general purpose planning systems) is an important and timely topic\n * In the empirical evaluation the authors have clearly made an effort to compare a wide range of contemporary LLMs that currently score well on leaderboards\n * The key concept behind the skill-mix framework is easy to understand", "weaknesses": "* The motivation for this evaluation framework needs further development. Sec 1. highlights accelerating rates of leaderboard saturation, training set contamination, and training corpora secrecy as pressing issues, and suggests 7 desiderata for new LLM evaluation frameworks; (a) relevant to general-purpose intelligence, (b) easy for humans to design and administer, (c) resistant to training-set contamination, (d) capable of revealing novelty in some sense, (e) easy to grade at scale, (f) easy to scale difficulty over time, and (g) comprehensible by the general public. This list seems promising, but the paper doesn't make it clear how skill-mix addresses each of these needs, or how these 7 desiderata are related to the 3 pressing issues with current evaluation methods. In particular, I see no reason to believe that, once released, skill-mix won't also suffer the problem of 'cramming for the leaderboard' (also see next point)\n * In the current form, skill-mix defends against 'cramming' by proposing a held-out private test set of skills and topics. However, any evaluation design that relies on private evaluation data is fundamentally flawed - the field of LLM evaluation design needs to learn from cryptography algorithm design on this count. This is also problematic from an open science perspective - making it harder for peer review to evaluate the efficacy of the evaluation framework for instance.\n * The theoretical justification for skill-mix is also unclear to me.\n    * For instance - does it make sense to combine /any/ $k$ of the N skills together, or are there some incompatible skill-tuples?\n    * The connections with pedagogy theory and the empirical observations from the Arora & Goyal 2023 paper are a good starting point, but it's not clear in what way combining skills relates to generalized intelligence. A clearer definition of 'skill' may help here.\n * Some 'late breaking' paper results are missing from the submitted version (e.g. footnote 3, Sec. 6) - the paper will be strengthened with the inclusion of these results in a future version.\n * (End of Sec. 5) \"Difference between model ranking on popular LLM leaderboards vs. SKILL-MIX provide evidence of \"cramming for the leaderboard\", further validating that SKILL-MIX is a good evaluation benchmark\" - the conclusion here doesn't follow from the premise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5wo8VwaXg4", "reviewer": "Reviewer_NFpe", "summary": "The paper presents a new evaluation benchmark SKILL-MIX for general-purpose language capabilities, specifically, a particular sort of compositional generalization. It tests the model’s ability to create text on-demand on a given topic and with a given combination of well-known skills. The paper provides an exemplary introduction and motivation for the task of appropriate evaluation of the newly emerging topics.  A key idea is that the skills and the topic are chosen randomly from a big list, which itself can be expanded in many ways to give new evaluations. Such evaluations may end up requiring the (Student) model to imagine and describe situations that does not exactly match anything seen during training. Grading this evaluation is shown to be possible with GPT-4 (and the authors spot-checked the grading). The authors also emphasized the important of human spot-checking, but since human evaluations were  found to have significant variance, they are not reported.", "questions": "1)Do you think that instead of have one strong evaluator, you can have a mixture of experts each capable to evaluate a single task exceptionally well?\n2) Can you please provide reference for \"While both models are creditable graders, they, like all current LLMs, were unreliable at simple arithmetic, which is important for calculating a total score.\"", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "I found this paper one of the best papers I reviewed recently (including the Neurips reviewing).\nI found the Introduction (Section 1) and especially \"Desiderata for next-generation evaluations\" extremely valuable for any LLM development. I am currently working on a task that requires complex evaluation of LLMs and the guidance and observations summarized in the introduction are right to the point of what we are looking for. \n\nI also find the idea of mixing random skills with evolving N as a way to mitigate some of the issues in current evaluations simply, yet elegant.  The authors also validate its effectiveness with numerous experiments.", "weaknesses": "1) I'd love to see some correlation of the presented results with human spot-checking and evaluation and if such correlation does not exist, discussion on what might be the reason\n\n2) I'd love to get some suggestion on how this framework can be extended beyond language to Vision - Language Models.\n\n3) I'm curious to see how the experiments with fine-tuning while releasing 10% of skills pan out.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5MBDIMQC2q", "reviewer": "Reviewer_VgUg", "summary": "The authors present Skill-Mix, an evaluation for LLMs which utilizes skill combinations to evaluate LLM capability on likely unseen tasks. They demonstrate that some of their results contradict existing results on popular LLM evaluations.", "questions": "See weaknesses section", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**********************Motivation:********************** One of the main motivations of the paper is good: LLM evaluations can be trained for so it makes sense to do some combinatorial benchmark which requires combining skills in a way that’s likely unseen in any training corpus.\n\n**************Method:************** The idea of skill combination as an LLM evaluation is good, and to the best of my knowledge (i’m not very familiar with LLM benchmarks), novel. It is definitely harder to expect knowledge of the benchmark to be contained in the training corpora of LLMs a priori.\n\n******************Dataset:****************** It’s good idea to release only 10% of the skills and topics for evaluation to prevent overfitting/training on the testing criteria.\n\n****************Results:**************** I’m glad the paper demonstrates that this evaluation is not so correlated with other existing benchmarks, as to prove that it is testing something different from the others with respect to LLM capability.", "weaknesses": "****************************Skill picking:**************************** Section 3.1 (and the appendix) describe how the authors pick the skills by hand based on textbooks on logical reasoning, rhetoric, and theory of mind. But what is the justification for using these types of skills? For example, i’m sure there are many skills not related to any of those topics which still require some sort of intelligent capability to combine meaningfully. The authors should explain this better.\n\n**********************Evaluation:**********************\n\n- One minor issue is that I don’t think the models picked necessarily need to be instruction-finetuned as regular, non-instruction finetuned language models are still able to follow instructions in the prompt. In fact, given that some recent work has demonstrated that instruction-finetuning on smaller models reduces their ability to produce as diverse of a response distribution, it could artificially hurt the performance of the selected open-source models in evaluation due to the selected topics occurring with a Google n-gram probability of ~1e-6 and them likely being fine-tuned on a much smaller dataset that may not include the selected topics/skills.\n- What is the variance among the automatically graded scores? It is stated that human variance is higher than the automatic ones, but neither seem to be presented anywhere.\n- The analysis of Skill-Mix vs other benchmarks could be more comprehensive — it would be interesting to see ******************correlation scores****************** against other benchmarks to see how skill-mix differs overall wrt individual LLM rankings.\n\n********Writing + Clarity + Motivation:********\n\n- Intro first paragraph: A missing “yet” between the 2nd and third sentence, otherwise these two sentences next to each other are a bit confusing.\n- Intro: “(including for some public models such as LLaMA-2)” this could be put next to “size of training corpora” as right now it’s directly after mentioning closed-source LLMs which makes the parenthesis seem confusing.\n- Intro: Why is a specific opinion from Hinton used as motivation for desiderata for the authors’ benchmark? While Hinton is an important figure in the field, a good justification should come from (1) an intuitive explanation of the problem, (2) consensus in the field for the problem, or (3) direct citation of evidence of the problem (or some combination of all 3). I don’t think citing a single person’s opinion is good justification in general or intuitive to a reader presented with a new problem.\n- Section 1.1: How does SKILL-MIX directly address the desiderata presented in the prior paragraph? For reader clarity, this question should be answered immediately after presenting the desiderata or else we will forget.\n- Section 1.1: Arora & Goyal is cited as demonstrating that scaling up language models → improvements at applying k’-tuples of basic skills, therefore motivating the authors’ Skill-Mix benchmark. But this doesn’t mean that applying k’-tuples of basic skills → better general purpose understanding. This is a problem with many LLM benchmarks, but still presents an issue that the authors should discuss in this paragraph.\n- Prior work: This section isn’t that well written. The authors should relate Skill-Mix to each of the types of prior work rather than describe them in isolation.\n- Section 4: This is a minor issue, but Section 4 can probably be made a subsection of Section 3 as it is part of the introduced SkillMix evaluation.\n- Table 1/2: Some sort of bolding on the best performing across each category for each k would be really helpful to parse all of these numbers\n- Section 6: Can the authors explain the derivation of the high probability statement for $O(T \\log T + N \\log N)$? Intuitively, it’s quite surprising given “random prompts” to expose these low-probability skills and their topics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)