Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving
Authors: Aniket Rajiv Didolkar
Anirudh Goyal
Nan Rosemary Ke
Siyuan Guo
Michal Valko
Timothy P Lillicrap
Danilo Jimenez Rezende
Yoshua Bengio
Michael Curtis Mozer
Sanjeev Arora
Venue: NeurIPS 2024 poster (2024)
Keywords: Metacognitive abilities of LLM
Mathematical Reasoning
Primary area: natural_language_processing

Abstract: \emph{Metacognitive knowledge} refers to humans' intuitive knowledge of their own thinking and reasoning processes. Today's best LLMs clearly possess some reasoning processes. The paper gives evidence that they also  have metacognitive knowledge, including ability to name skills and procedures to apply given a task. We explore this primarily in context of math reasoning, developing a prompt-guided interaction procedure  to get a powerful  LLM to assign sensible skill labels to math questions, followed by having it perform semantic clustering to obtain coarser families of skill labels. These coarse skill labels look interpretable to humans.

To validate that these skill labels are meaningful and relevant to the LLM's reasoning processes we perform the following experiments. (a) We ask GPT-4 to assign skill labels to training questions in math datasets GSM8K and MATH.  (b) When using an LLM to solve the test questions, we present it with the full list of skill labels and ask it to identify the skill needed. Then it is presented with randomly selected exemplar solved questions associated with that skill label.  This improves accuracy on GSM8k and MATH for several strong LLMs, including code-assisted models. The methodology presented is domain-agnostic,  even though this article applies it to math problems.

Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving
Aniket Didolkar ', Anirudh Goyal ', Nan Rosemary Ke 4, Siyuan Guo 3,5 Michal Valko 4, Timothy Lillicrap Danilo Rezende Yoshua Bengio ', Michael Mozer 4, Sanjeev Arora
Abstract
Metacognitive knowledge refers to humans' intuitive knowledge of their own think- ing and reasoning processes. Today's best LLMs clearly possess some reasoning processes. The paper gives evidence that they also have metacognitive knowledger including ability name skills and procedures to apply given task. We explore this primarily in context of math reasoning, developing procedure to get powerful LLM to assign sensible skill labels to math questions, followed by having it perform semantic clustering to obtain coarser families of skill labels. These coarse skill labels look interpretable to humans.
prompt-guidee interaction
To validate that these skill labels are meaningful and relevant to the LLM's reason- ing processes we perform the following experiments. (a) We ask GPT-4 to assign skill labels to training questions in math datasets GSM8K and MATH. (b) When using L.M to solve the test questions,
present it with the full list of skill
labels and ask it to identify the skill needed. Then is presented with randomly selected exemplar solved questions associated with that skill label. This improves accuracy on GSM8k and MATH for several strong LLMs, including code-assisted models. The methodology presented is domain-agnostic, even though this article applies it to math problems.
Introduction
Large language models (LLMs) have demonstrated remarkable advancements in recent years at natural language inference tasks [1-7], as well as scientific and mathematical problems [8-11], although their limitations on mathematical problems are also well-documented [12-17].
core concept in human pedagogy is Metacognition [18], sometimes described as thinking about thinking. It refers to ability to reason about one's own cognitive processes as well as about learning- relevant properties of information mulated knowledge of this type. Pedagogy research shows that improving learners' metacognitive knowledge can improve their capabilities, for example on math [19, 20]. The current paper raises the question "Do LLMs also have metacognitive knowledge?" And if yes, Can we bootstrap such knowledge to further improve LLM capabilities?
data. Metacognitive Knowledge refers the learner's accu-
At first glance, this quest seems difficult. Deciphering LLMs' inner working from their huge set of parameters all results of lonerr optimization- notoriously hard. Furthermore, scientists lack parameter access to most leading AI models. But there are still reasons to hope we can understand metacognition by interacting with LLMs. They display some human ties, such ability improve their math reasoning via Chain of Thought (CoT) [21] and also the "Let's think step by step"
01 Mila, University of Montrcal, Princcton University." 3 The University of Cambridge, Google DeepMind Max Planck Institute for Intelligent Systems Corresponding authors: adidolkar
2tiidolkar123@gma1.... 230gmail com, anirudhgoyal91199gmmlll
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Creating Skill Exemplar Repository
Peonce Skill Clustering
Skill Labelling
Exemplars
Pd Fcanctioe
9k111-1
9d Ice
14111-2
LLMA
24011-4
Drs Quaetios 3
SKILL EEC
Inference
Inference Skill Labelling
Select In- In-Context Exemplars
Compute Answer
(Test Question)
LLMA
Skill-11
LLMB
Creating Skill Exemplar Repository: First, an LLM labels each question with corre- sponding skill, as detailed in the prompt provided in Appendix Figure 2 (left). Next, the LLM is asked to combine similar fine- grained skills into broader skill clusters, which represent complex skills. This greatly reduces the number of unique skills from the first stage. The prompt for this is depicted in Appendix Figure 2 (middle). Then the LLM is asked to reclassify all examples from the training set into one of the post-clustering skills. Using these we create a 'Skill Exemplar Repository' which consists of skill exemplars consisting of skill names and their corresponding question/answe examples. Inference: During inference, we use an LLM to first label a test question with one of the skills from the skill exemplar repository. Next, we fetch exemplars from the repository with the same skill and provide them as in-context examples to solve the test question.
Figure 1:
prompt [22]. These were generally perceived as convenient tricks to get around the limitations imposed by the LLM's auto-regressive nature. But other pieces of evidence have emerged about existence of LLM metacognition. A notable example is Ask-LLM [23], whereby the LLM appears to give surprisingly helpful answers to the question "Is this a good training datapoint for an LLM?" current paper reports on similar direct approach to deciphering LLM metacognition: Just go ahead and ask it!
Specifically, the Metacognitive Knowledge of interest in this paper is the catalog of skills (from the LLM's viewpoint) that it applies while solving math questions. Pedagogy research has uncoverex a rich catalog of skills in humans, ranging from simple ones operations on variables, solving equations, grasping the concept of function- to difficult ones such as grasp of difficult theorems and proof strategies. But currently mathematical datasets used in LLM research (such as MATH [16]) partition problems using broad human-assigned topics such as "probability" and "algebra." We are interested in a more fine-grained understanding of LLM skills.
Skill Discovery: Our automated approach for the discovery of skills utilizes trate-of-ter oo-the--rtttt LLMs to identify their own catalog of math skills and then organize datasets using that catalog. Stage of our methodology involves instructing the LLM assign skill labels each example within given dataset. Usually this results in fine-grained skills, and too many skill labels. In Stage 2, the same LLM is asked perform semantic clustering the labeled data, grouping examples by the similarity of their underlying skills (as perceived by the LLMM). Each resulting cluster represents a larger set of examples. Our method retains only these
more soarse-grained skill that is applicable coarse skills. (To give an example, for the MATH dataset, Stage identified approximately 5000 skills, which Stage 2 reduced to 117 coarse skills.) A random subset of examples representing the coarse skills are retained as its skill exemplars. (See Figure I and Appendix 10). To subsequently improve (in-context) math problem solving by LLMs we use the repository of skill exemplars -each labeled with coarse skill. Here the LLM is given a new question and the above list of coarse skills and asked to identify the skill needed to solve this new question. Then the LLM provided the previously identified exemplars for the selected skill as in-context examples to guide its problem-solving. We note that this is reminiscent of how human problem-solving is taught by presenting examples very congruent with the specific problem at hand. Here we find that LLM problem solving improves

---

Dataset
Topie
Skills
GSMBK
muliphication and ashition. buthee
unihmelic..pper maloapice pencenaagesaalccasiiiin subtraction, algebra, cubrration.mmi.lliiii millippicaiiiiiommm divisiiis, multiplicttion and jubtractionn adliiionjinl_uueen sutenntioo,, precmtage_sallalann addittionssueemntti subeesttion, aven-
ape.calculation subraccio.mmulippiaaiii division, additin. lquarreuuations. algebraic zeassninng
Table 1: List of Skills for each Dataset This table lists down the skill obtained after the skill clustering phase for each dataset and corresponding topics. Skill names were provided by GPT-4- 0613. The skills of the other topics in MATH can be found in Appendix Table 10
using the skill labels and skill-exemplars provided by an LLM on the same dataset. This provision of skill-exemplars can be seen as a new addition to on top of known prompting methods such as fhain-of-thoughy
Although describe method only in context of math, it seems general enough to be broadly applicable to problem-solving of other sorts. This is left for future work.
Paper organization and main results: Section 3 describes the method and Section 4 describes experiments. Using strong L.LM- GPT-4 - identify skills, we validate the usefulness of these skills by demonstrates significant 1 11.6% enhancement over CoT on the MATH Dataset using the method described in Section Furthermore, the identified skills also improve the generation of code-based solutions for the problems within the MATH dataset giving 7.52% improvement over the baseline PAL approach [24], which also instructs the model to generate code. Section 4.3 shows that the the skill exemplar repository created for MATH noticeably improved in-context performance for weaker LLMs on the same dataset and that the repository for GSM8K helped improve in-context performance for other math datasets. This shows that powerful LLM can be used for deeper understanding of skills that translates across other L.MSs and related datasets.
2 Related Works
For human learning, statistical methods can infer latent skills from data and use the inferred skills to more accurately forecast student learning [25, 26]. In machine learning, works that study learning via skill induction include [27-30]. These start with some definition of skills in terms of model parameters, whereas we use a powerful L.M in black box way to identify and consolidate skills. A discussion of various prompting strategies is covered in Section 4 and Appendix Section 9.
3 Automated Skill Discovery
We descrihe an automated process for categorizing mathematical questions according to specific skills needed to solve them. See Figure 1. Recent works relating skills and LLMs [31, 32] were an inspiration. Conceptually, the strategy involves the creation of a detailed skill exemplar repository, which contains compilation of skill names alongside respective illustrative examples (comprising both questions and answers). During the inference stage, when presented with a question, the LLM initially looks among skill exemplars identify the skill that is best suited for the question. The LLM then utilizes the corresponding exemplars for that skill as in-context prompts.
The proposed setup consist of training set T {96 az)). (oT af)..... ( oT), where of and a are question and answers from the training set. The training set is used for selecting in-context examples for inference. Our test set also consists of set of questions and answers E := ((g af), ). (gf , ( a )] To create the skill exemplars, we first label the training set, T, with skill per example using LLM. Next, we label the test set with skills to retrieve in-context examples with matching skills from the skill exemplar repository. The exact procedure of labelling the training and test set with skills is different and detail both approaches below.
Notation.
3.1 Skill Labelling: categorizing mathematical questions according to specific skills
The process is illustrated in Figure 1. It had the following steps.
Assign Skill Name for every example in training Set T: Using a carefully curated prompt (given in Appendix Figure 2 (left)), we asked LLM to label each training instance with single skill name and reason for that assigned skill. Figure (top) represents this process. Applying strong LLM for this task - GPT-4-0613- we found that for the 7, 000 instances the GSM8K dataset [33], it specified approximately 500 unique skill names. For the 7, 500 examples in the MATH dataset [16], it specified
3

---

5, 000 skill names. (This perhaps reflects the hardness and diversity of MATH compared to GSM8K.) Although these skill labels precisely encapsulate the capabilitiiss requisite for solving each question, it is clear that the granularity is excessive, raising issues reminiscent of classical "overfitting."
For example,
for the question "In a triangle, the area is mumerically equal to the perimeter. What is the radius of the inscribed circle?
(A) 2 (B) (C) (D) 5 (E) 6" GPT-4 came up with the skill name understand- ing. of Atttarraeeppaat properties and _ircle radius calcullaion..
Despite descriptive accuracy, its high specificity may limit its utility, as it is improbable that an identical question embodying this precise skill will recur. To address this, the initial labelling phase is followed by a phase of skill clustering, aiming to generalize the skill categories for broader applicability.
Semantic Skill Clustering: In this phase, the LLM was prompted to aggregate the skills identified
in the skill labelling stage, specifically to group similar skills into broader categories (Figure 1 (top)) and assign descriptive label to each category. (The prompt appears in Appendix Figure (middle). ) Again utilizing GPT-4-0613 for this, we obtained reduced skill set comprising of 22 skills for GSM-8K and 117 skills for MATH. The list of skills are presented in Table I, and Appendix Table use the LLM to reclassify all examples in the training set T using these new skill names from the clustering phase. Thus the initial highly detailed skill labels get consolidated into broader, more universally applicable categories. For instance, the question initially labelled as "understanding of triangle properties and circle radius calculation" is relabeled to have the skill name "understanding. of triangles". This modification significantly enhances the applicability of the training set for a wider range of problem-solving scenarios.
10. Subsequently,
Skill Exemplar Repository: Following the skill clustering and relabelling process of the training set, we established a 'Skill Exemplar Repository.' This contains a curated selection of skills and their corresponding exemplars, specifically questions and answers, derived from the training set T. The structure of the skill exemplar repository is formalized as follows: skill exemplar repository = (so. 90' ag)) ($1, of at ) ...., (Sp, q. a ). where s; denotes the skill label associated with the i-th question-answer pair (g. a.)) See Figure (top) for an example of such a repository. This systematic compilation facilitates efficient referencing and application of relevant examples corresponding to specific skills during inference. App. Tables 11, 17, and 18 illustrate examples from the skill exemplar repositories for the GSM-8k and MATH datasets respectively created using GPT-4-0613. We that each question is labelled with a human interpretable and intuitive skill name.
3.2 Inference at test time
In the testing phase, the LLM is given a math question Q. It is asked to first select one skill from the list of skills in the repository, say 8; that is most relevant to the question. (The prompt employed for this step appears in Appendix Figure 2 (right).) Next, K Exemplars corresponding to 8,, randomly picked, are then employed for few-shot prompting as usual. By providing the LLM with contextually relevant, skill-specific examples from the repository, one expects to enhance its effectiveness at answering the question Q. This process is depicted in Figure (below).
Transferring skill exemplars to other datasets The broad range of questions, answers and skill labels in the exemplar repository makes it an attractive source of relevant in-context examples for solving various mathematical problems. To demonstrate such adaptability and utility we applied the Skill Exemplar Repository derived from GSM8K dataset to solving various existing math word problem datasets that were designed to evaluate concrete mathematical skills or concepts. Section 4.3 reports notable improvements in problem-solving capabilities across domains.
3.3
Skills from strong LLMS improve weaker LLMs
Through the methodology described above, we find that strong LLM such as GPT-4-0613 is able assign intuitive and human interpretable skill names questions. These skills are representation of the metacognitive knowledge of the LLM. We consider whether this knowledge can be applied to other LLMs specifically weaker L.Ms. Section 4.3 shows that skill-based in-context examples, labeled using a stonger L.M as described earlier, also significantly enhance the performance of less advanced models, such as Mixtral [34]. This underscores that the skill-based knowledge categorization from one LLM is broadly applicable to other LLMs too.

---

Skill-exemplars improve various prompting methods Our approach is designed to be synergistic with a range of prompting techniques, thereby offering broad upplicability across various method- ologies. It can be seamlessly integrated with numerous existing prompting strategies, including the Chain of Thought (CoT) approach [21]. PAL [24], and the self-consistency method [35]. In each of these instances, the proposed method enhances the existing framework by substituting the con- ventional in-context examples with those meticulously selected from the Skill Exemplar Repository. This integration not only preserves the inherent strengths of the original prompting techniques but also augments them by leveraging the specificity and relevance of the skill-aligned examples. This adaptability underscores the versatility and potential of the proposed approach to improve the efficacy of various language model prompting strategies.
Prompting CoT
Pre-Algebra Geometry Inter-Algebra Algebra Probability Pre-Caleulus Nom. Theory Overall
42.2
Complex CoT
71.6
36.5
23.4
70.8
53.1
26.7
49.6
50.30
Col' + Topic-Based
71.16
39.45
24.14
67.90
54.64
31.13
47.03
50.31
Coll+ Skill-Based
74.28
41.75
27.02
73.12
58.01
33.70
53.88
Table 2: Text-based prompt results on the MATH Dataset. Our Skill-Based approach, employing CoT prompting, demonstrates superior performance over all other methods across all topics within the MATH dataset. All experiments were conducted using GPT-4-0613.
4 Experiments
In Section 3, we have described a procedure to extract metacognitive knowledge from LLMs in the form of skill annotations for mathematical questions. In this section, we show that this knowledge of skills can be further used to improve reasoning in LLMs by using them to provide pertinent in-context examples for solving new mathematical problems through the process described in Section 3.2 and depicted in Figure 1 (below). Our evaluation focused on three distinct areas: Text-based Prompts: We utilized chain-of-thought prompting, as detailed in Section 4.1. This method involves providing step-by-step reasoning in the prompt guide the model's thought process, Program-basel Prompts: Here, we employed program-aided language models (PALs), described in Section 4.2. PALs integrate programming logic within the language model, aiming to enhance its reasoning capabilities, and Transferabilit: We investigate the generalizability of these skills across different LLMs and datasets, as elaborated in Section 4.3. This aspect tests how well the skills transfer different LLM models and unseen datasets. Our results demonstrate that knowledge of skills significantly improves performance for both text-based and program- based prompting across different datasets. Furthermore, these skills exhibit strong transferability, boosting mathematical reasoning capabilities across other maths datasets and I.IM models. We also conduct a detailed analysis to gain a deeper understanding how our approach influence the reasoning abilities of LLMs. Finally, we present an initial exploration into labeling each question with multiple skills instead of single skill followed by an experiment which demonstrates the flexibility of the proposed approach by applying it for alignment.
Prompting Methods We investigate two prominent types of in-context prompting methods for enhancing mathematical reasoning in LLMs: Text-based Prompting: Utilizes text examples to demonstrate
steps, with Chain-of- Thought (CoT) [21] being a prime example. Program-aided Prompting: Employs programs to showease reasoning steps, as seen in Program- -iddd Language Models (PALs) [24]. To assess the effectiveness of these methods, we replaced the standard in-context examples used by CoT [21] and PAL [24] with examples from our skill exemplar repository. We then evaluated the performance of LLMs with both text-based and program program-based prompting, using our skill exemplars versus standard examples.
Baselines Our evaluation also includes a comparison with four baselines to isolate the impact of our skill-specific examples: Random: This baseline randomly selects examples from our repository in contrast CoT's fixed examples, highlighting the necessity of skill-aligned example selection. Topic-Based: Examples are grouped by broader mathematical topics (e.g.. algebra), as in the MATH dataset [16]. This tests whether finer-grained skills (as detailed in Table 10) offer an advantage over broader topic categorizations. ComplexCOT [36]: Chooses complex in-context examples for CoT, allowing us to analyze whether complexity or skill-specificity has a greater impact on performance. Retrieval-Rit -RiDevall--DDD [37]: This selects relevant in-context examples for few-shot tasks similar to the proposed approach. They first map the examples to latent space and then selects top-k in-context examples based on cosine similarity to the example. Through these comparisons, we aim to discern
5

---

the relative benefits of skill-specificity and complexity in example selection for enhancing LLMs' mathematical reasoning capabilities.
Datasets We evaluate the proposed approach using a variety of mathematical reasoning datasets. We start with the GSM8K dataset [33], which comprises grade-school level math problems. We then move on to the challenging MATH dataset [16], known for its competition-leve problems.
To examine the transferability of skills, we apply the skills from the GSM8K dataset to other math word problem datasets. These include SVAMP [15], ASDIV [38], and the MAWPS suite (SingleOP, SingleEQ, AddSub, MultiArithh [39]. Each dataset presents its unique set of challenges and complexities, allowing us to thoroughly assess the adaptability and effectiveness of our approach across different mathematical contexts. For details about these datasets, please refer to the Appendix 10.1.
Language Models In Section 10.4 of the Appendix, we conduct comparative analysis of GPT- 4-0613, GPT-3.5-Turbo, and Mixtral-8x77 in their proficiency in generating precise skill labels. Through experimentation, we show that the skill labels annotated by GPT-4-0613 lead the strongest in-context learning performance on the MATH dataset [16]. Therefore, we establish GPT-4-0613 as the primary model for skill labeling, clustering, and conducting the majority of our experiments. For transfer experiments, as outlined in Section 3.3 and further detailed in Section 4.3, we evaluate the performance of the Mixtral 8x7B model [34]. This dual-model approach allows us assess the effectiveness of our methods across different advanced language models.
4.1 Text-based Prompts
We consider the GSM8K dataset [33], containing grade-level math word problems, and the MATH dataset [16], featuring competition-level math problems. These experiments aim to assess the efficacy of our approach across wide range of mathematical complexities, specifically using text- based prompting strategies. All experiments were carried out using GPT-4-0613, employing 8-shot prompting and a decoding temperature set to 1.0.
Results on GSMSK. GSM8K dataset [33] contains 7.5k training problems and Ik test problems. The skill exemplar repository is created using the training data only, refer to Section 3.1 for details. See Table I in the appendix for examples from the skill exemplar repository.
Base Model
Prompting
GSM8K
G
Retrieval
76.8
CoT + Skilll-asee
82.03
CoT
93:00
Col Random
92.87
We utilize the skill exemplar repository solve test set problems from the GSM8K dataset, as outlined in Section 3.2. The results are shown in Table 3. Our Skill-Based approach outperforms both the Chain-of-Thougl (CoT) and Random baselines on the GSM8K dataset, underscor- ing the importance of accurate skill assignment and per- tinent in-context examples in effective problem-solving Furthermore, augmenting the Skill- approach with self-consistency (SC, presented as maj@ in Table 3) tech- niques [35] leads to even better performance, highlighting the adaptability and effectiveness of our method. For the SC experiments, we sample 5 reasoning chains from the LLM and choose the most frequent answer. Additionally, we provide detailed breakdown of per-skill accuracy for both the proposed approach and the Random approach in Ap- pendix Figure 3. To further emphasize the effectiveness of the proposed method, we compare it to the Retrieval-Re RSDD approach [37] which is also a pertinent in-context example selection approach for few-shot prompting. The results are presented in Table 3 show the superiority of our approach as compared to the Retrieval-RST approach. We use GPT-3.5-Turbe backbone for this comparison.
GPT-4-0613
Col's + Skill-Based
94.31
CoT Skill-Based (maj@5)
95.38
Table 3: Text-based prompt results on the GSM8K Dataset. Our Skill- Based approach outperforms various other methods the GSMSK dataset across two different models: GPT-3.5 Turbo and GPT-4-0613. Refer to text for description of baselines.
Results on MATH. The MATH dataset, comprising competition-level math problems, covers topics like Pre-Algebre Algebra, Intermediate Algebra, Geometry, Number Theory, Precalculus, and Probability. Its training set has 7.5k examples and the test set has 5k examples, each labeled by their respective topics. Following the methodology described in Section 3. we created a Skill Exemplar Repository using the MATH dataset's training set. This repository is showcased through examples in Appendix Tables and 18, providing insights into the range and nature of skills covered in the MATH dataset. Furthermore, in Appendix Table 12 show examples of the relevant in-context

---

Prompting
Inter Algebra Precalcalus Geometry Num. Theory Prohability FreAlgehra
PAL (4-sho))
J0.9
23.2
31.7
66.1
57.9
73.2
65.3
52.0
PAL
Skill-Based Code (asee)
Sieill-Based (7 Skill-Based Code-Based)
41.46
73.39 62.00
Table 4: Program-aided prompts results on the MATH dataset. This table illustrates the per- formance achieved by employing the tasks drawn from the MATH dataset using GPT-4-0613, Evidently, supplying pertinent in-context examples grounded in specific skills enhances the program generation performance of GPT-4-0613, leading to a notable improvement across all topics encompassed in the MATH dataset.
approach to generate code for problem-solving
Prompting
SC
Fre Algehra
Geometry
Inter- Algehra
Algebra
Probab- lllty
Fre- Calculus
Num. Thewry
Overall
Emaj
maj
"Topic-Bi Based
x
42.94
17.33
11.30
40.78
19.85
14.47
16.3
26.14
Skill-Resed
47.76
19.42
13.29
43,05
20,04
16.12
18333
28.44
-Topic
2025
15.73
2033
30.75
Skill-Based
maj
21.48
32.44
Table 5: Transfer Skill Exemplars to Other Models. All experiments are performed using the MATH dataset the Mixtral 8 x 7B model, comparing against standard CoT, CoT with topic-based exemplars, CoT with skill-based exemplars, CoT with self-consistency (maj@4) using both topic and skill-based exemplars. Skill labels and exemplars obtained from GPT-4-0613, 13. The enhanced performance of Skill-Based indicates effective transferability of skills from GPT-4 to another model.
examples selected from the skill exemplar repository to solve a given question. We can see that selected exemplars are similar to the question to be and correctly illustrate the concepts required by the question.
Results on the MATH dataset are shown in Table 2. For this analysis, our proposed approach utilizes straightforward Chain-of-Thonn ohouhttt (CoT) method, wherein the in-context examples sourced from the skill exemplar repository. Our method achieves a notable improvement in performance, surpassing the standard Chain-of-Thought (CoT) by an impressive 11.6%. We also outperform 3.5% over Complex CoT, and 3.5% over the Topic-Based approach. These results highlight the efficacy of our approach, particularly with its fine-grained skill labeling. The fact that it surpasses Complex CoT is especially noteworthy, indicating the importance of selecting in-context examples that are highly relevant to the specific problem, rather than using their complexity as a proxy for quality.
4.2 Program-Aided Prompts
Program Aided Language Models (PALs), as developed by 24, are designed to produce program- based solutions for problem-solving, employing Python the programming language. This approach addresses the issue of logical or calculation errors in Language Models by translating solutions into code, thus leveraging compiler for final answer generation. In our integration of Skill-Based prompting with PALs, we modify the in-context example structure: we use three non-code- based examples from our skill exemplar repository based on skill matching, followed by one fixed code- based example, totaling four in-context examples. The specifics of these prompts are detailed in Appendix Section 10.5.
Our experiment, results of which are shown in Table 4, tests this modified approach on a subset of 500 examples from the MATH test set, same those used in [40]. The findings are significant: despite only one code-based in-context example (compared to PAL's four), our approach shows a 7.52% improvement over PAL. This highlights the impact of strategically chosen, relevant in-context examples, on the accuracy of code generation for problem-solving.
4.3 Transfer of Skill Exemplars
Studying Transfer of Skill Exemplars to Other Models In this study, we explore the transferability of skills from GPT-4 to other LLMs, particularly focusing on Mixtral 8x7B [34]. This experiment replicates the setup used for the earlier experiments with GPT-4-0613, utilizing the same skill exemplar repository and skill labels for the MATH dataset test set examples initially labeled by GPT-4-0613. For each problem, 4 in-context examples are chosen based on skill-matching and outputs are sampled with decoding temperature of 0.2. The results are displayed in Table 5. We use 1 A100L GPU for this experiment.
7

---

Example
Example
Question
Compute sin 210".
Find
solve problens, have know (he urgunometric functions have period 360", which means sin 210" sin (210"
number, henze it connot be ovaliated. The ceiling fine- rion applicahle only real mimbers therefere expression undefined.
The calhe
25
Topic-Based Solution
180") sin 30 1/2.
The angle 210" is 30 below the negative 2200 sin(-30") 30%)
The cube root of 25, 2- equals 2.92 up prosinatelyy since the colling function up nearest integer greater namber, the value will he -2 Skill Used:
Skill-Based Solution
-1/2. Skill Used: trigonometry.
given.
exponent. and skilss
Table 7: Instances Benefiting from Skill-Based Approach This table illustrates instances where our skill-based approach empowers the Language Model (LLM) to apply relevant skills effectively. hed-highlighted text reveals conceptual errors by the Topic-Based baseline, while blue-highlighted text showcases skillful and accurate skill application.
Here, we compare our Skill-Based approach against two baselines: Chain-of-Thought with self- consistency (SC) as per [35] and the Topic-Based approach. For implementing self-consistency, we generate four reasoning chains and select the most frequent answer (noted as maj @4 in Table 5). The results demonstrate that our Skill-Basch approach surpasses both the Topic-Based and CoT approaches. Notably, our approach, even without confissnnss, matches the performance of CoT with SC, highlighting its efficacy in extracting correct reasoning paths and concepts. Furthermore, when combined with self- cofissnssstenny, our approach shows a remarkable 4.0% improvement over CoT with SC, affirming its superior efficacy in skill application and reasoning.
Studying Transfer Skill exemplars to Other Datasets Here, we investigate the trans- ferability of skills from the GSM8K training dataset to other math word problem datasets. We apply our approach to various datasets, in- cluding SVAMP [15]. ASDIV [38], SingleOP, SingleEQ, AddSub, and MultiArith [39], each comprising distinct problem types. We utilize the GSMsk-derived skill exemplar repository for these datasets, testing skill transferability across similar datasets. Notably, we use the pre- clustering skill labels, as these datasets feature finer granularity problems compared to GSM8K, making post-clustering skills less effective.
Frompting
WYAMP|SingLOPP
SingleEQ
AddSub
MultiAriih ASDIV
CoT
91.9
97.2
97.2
93.9
98.0
92.7
PAL
92.2
95.2
96.8
94.9
98.5
90.2
COT PAI.
93.7
97.3
98.6
95.7
99.0
93.5
Col Skill Skill-Bosed
92.6
97.86
94.03
Table 6: Transfer of Skill Exemplars to Other Datasets Investigation of skill transfer from GSM8K to different math word problem datasets using GPT-4-0613. Questions in the target dataset are labeled with corresponding skills from GSM8K, and in-context examples based on skill-matching. The proposed approach achieves the highest accuracy 4 out of 6 cases.
selected
The results, presented in Table 6, demonstrate the effectiveness of our approach. We employ a CoT-based method with 4-shot prompting and greedy decoding, aligning with the baseline settings. Our -kill-Based approach consistently surpasses the base CoT across all datasets. We also benchmark against PAL-based approach and hybrid CoT + PAL approach from [41]. where the model outputs both CoT and PAL solutions and selects the most accurate. Our Skill- Basle approach outperforms CoT + PAL in 4 out of 6 datasets, offering simpler yet more effective solution. These findings affirm the potential of skill knowledge transfer from one dataset to other similar datasets.
4.4 Analysis
We delve into the impact of Skill-Based precise concept and skill application, Firstly, we pinpoint successful instances where Skill-Based prompts guide the LLM in selecting and applying the correct skills. Secondly,
investigate cases where, despite pertinent Skill-Based prompts, the LLM fails to utilize the right concepts. Lastly, we quantify these instances of failure and compare them against baseline models, assessing the efficacy of Skill-Based prompting in enhancing the LLM's performance. All experiments are performed using GPT-4-0613.
Instances of LLM benefiting from Skill-Based Approach In Table 7, we compare the effectiveness of the Skill- Based approach against the Topic- Topic-Based approach in problem-solving scenarios through examples. The -ased approach significantly improves the model's 's reasoning and skill applica- tion. We highlight the reasoning errors of the Topic-Base Based approach in red and the correct reasoning steps undertaken by the Skill-Based approach in blue.
8

---

Our analysis reveals that the Topic-Band Srpic-Basedde approach misapplies essential skills. For example, Table 7 shows a fundamental misunderstanding of trigonometry in Example and fails to recognize negative cubes in Example 2. These errors are notably absent in the Skill-Based approach, demonstrating its superior understanding and application of key concepts.
Occurrences of Incorrect Answers Despite Employing Skill-Based Approach We examine the limitations of the skill-based approach in Table 15 (appendix). This table highlights instances where the model, despite using a skill-based approach, fails to produce correct answers. We use blue to denote correct reasoning steps and red for errors.
In Example 1, both the Skill-Based and Topic-Based Based approaches correctly apply the logarithm formula but err in selecting the appropriate number or imput, categorizing this error as "main skill error" or "skill error." This demonstrates a failure in correctly applying the primary skill needed for the question, highlighting limitation of the proposed approach. Example 2 further illustrates this limitation. Although the Skill-Based approach correctly uses counting concepts, erroneously calculates the number of diagonals in a hexagon. This error indicates shortfall in the application of certain secondary skills required to solve the problem such as, in this instance, understanding properties of hexagon.
These examples suggest that while the Ball-B approach effectively guides the application of the main skill required for question, it may falter in the application of secondary skills or in the comprehension of specific question properties. This analysis underlines the approach's 's strengths in primary skill application but also its limitations in more nuanced compound skill scenarios. It would be worthwhile to work with more complex skills.
Additional Metrics We introduce three metrics to evaluate the effectiveness of the proposed approach, using examples from the MATH dataset and employing GPT-4-0613 for classification. These metrics are: MAIN SKILL ERROR (SKILL ERROR): This indicates failure in understanding or applying the primary skill required for a question, SECONDARY SKILL ERROR: This denotes errors in comprehending or applying secondary skills necessary for the question, CALCULATION ERROR: This reflects mistakes in the calculation process during pnestiin--olvingg
These error types are not mutually exclusive; single instance may exhibit multiple error types. Correctly solved instances show none of these errors. GPT-4-0613's role in classifying examples into these categories is detailed in Appendix, Section 10.7, and its effectiveness is evidenced by the classifications in Table 15. To calculate the metrics, we first determine error rates for each error type and then derive success rates. These rates indicate how often the model correctly applies main and secondary skills, as well as performs calculations, across various questions.
Appendix Figure 4 displays the SKILL SUCCESS RATE, SECONDARY SKILL SUCCESS RATE, and CALCULATION SucCESS RATE for both Skill-Based and Topic-Based Based approaches. We expect the skill-based in-context example selection to be useful for reducing main skill errors. Our hypothesis supported by our findings, which show a higher SKILL SUCCESS RATE for this approach. This suggests that the model more frequently uses the correct skill with the Skill-Based approach compared to the Topic-Base Based baseline. Additionally, the proposed approach also demonstrates effectiveness in reducing secondary skill errors and calculation errors, underscoring its overall superior performance.
4.5 Multiple Skills
As demonstrated in the previous section, labeling exemplars with single skills poses challenges for questions that require multiple or compounded skills. In this section, we conduct a preliminary investigation into an approach which labels each exemplar with multiple skills.
We consider the MATH dataset for this experiment. We create the Skill Exemplar Repository in three steps. First, for skill labeling, modify the prompt shown in Figure 2 (left) to instruct the model to output multiple skills required to solve each question. Next, we perform skill clustering by passing the list of skills to the LLM and instructing it to combine common skills into representative skills. This clustering process is repeated iteratively until there are a total of N skills in the repository, where N is hyperparameter set 150. Finally, in the skill relabeling step, reassign all questions with skills from the clustered list, ensuring each question is labeled with multiple skills.
During inference, we label the inference questions with multiple skills derived from the clustered list of skills in the repository. We then fetch K in-context examples from the Skill Exemplar Repository that have the most skill overlap with each inference question.

---

We present the results in Table 8. We adopt the same setup as Table 2, which uses GPT-4-0613
and 4 in-context examples. We can see that the multiple skill approach outperforms the single skill approach thus showing its strong potential.
Prompting
Pre-Algebra
Geometry Inter-Algebra
Algebra
Probability
Pre-Calculus Num. Theory Overall
CoT Skill Based
74.28
41.75
27.02
73.12
58.01
33.70
51.10
$3.88
CoT --kil---asses Based (maltiple skills)
45.93
71.01
53.38
38.09
49.07
55.14
Table 8: Multiple Skills for MATH. In this table, we investigate labeling each question with multiple skills. We find that with multiple skills the proposed approach outperforms the single skill approach thus demonstrating the strong potential of multiple skill labeling.
4.6 Metacognitive Abilities beyond Math
We extend the proposed methodology for the problem of alignment via in-context learning [42]. We use the just-eval dataset introduced in Lin et al. [42] for this experiment. To apply the proposed approach in this setup we first curate a skill exemplar respository of 5000 examples from the alpaca dataset [43], 1000 examples from the lima dataset [44], and 5000 examples from hh-rlhf red team dataset [45] using the same prompts mentioned in Figure 2. Next, we label the examples in the just-eval dataset with skills from the skill-exemplar repository using the prompt shown in Figure 2 (right). We present examples from the skill exemplar repository in Apendix Section 10.9.
helpluiness
clarity
factuality
depth
engagement
safety
Col'+ + Kandom
3.61
4,33
3.77
2.35
2.90
3.65
CoT + Skill Based
3.73
4.40
3.89
2.64
3,01
3.78
Table 9: Alignment. We apply the proposed Skill-Based approach for task of alignment via in-context
learning. We find that proposed -kiledBased approach outperforms the Random approach.
Next, for answering each question in the just-eval dataset, we retrieve 3 in-context examples of the same skill as the question. For the baseline, we sample random examples from the skill-exemplar repository. We use the Mistral-7B [46] for these experiments. We report the metrics introduced in Lin et al. [42]. Each of these metrics are computed by prompting GPT-4 rate the LLM answer with score from 1 to 5 based on the metrics mentioned in each column. The final score is calculated as the average score across all samples. We present the results in Table 9. We find that the proposed approach outperforms Random on alignment task thus showing its effectivness in domains beyond math.
Discussion and Conclusion
We presented framework for extracting metacognitive knowledge from Large Language Models in the form of skills that categorize questions in mathematical datasets based on concepts required to solve them. This led to Skill Exemplar Repository, containing list of mathematical question- answer pairs annotated with the respective skills needed (in the LLM's own estimation) for the solution. Leveraging this repository, we furnish pertinent in-context examples to Large Language Models (LLMs) for tackling previously unseen mathematical questions. Our experiments show substantial empirical enhancements across diverse mathematical datasets, ranging from grade-level math problems to intricate competition-leve math challenges. The enhancements in performance via use of the repository also transfers to weaker L.MSs
One limitation of our methodology is that it assigns only one skill to each math question. As discussed in Section 4.4, mathematical problems often require combination of primary skill and various secondary skills. We leave design of a more advanced approach -say, using an LLM to create hierarchies of skills to assign multiple skills to each datapoint- for future work.
While this paper primarily addresses in-context learning, our future goal is to extend these method- ologies to improve all models through fine-tuning processes. Presently, our framework relies on the availability of advanced models like GPT-4. However, the skill discovery process improved in-context learning for GPT-4, which suggests that using skills to fine-tune GPT-4 may raise its capabilities. This hints more broadly at a path towards bootstrapping model capabilities -and not just math that seems worth exploring.
10

---

6 Acknowledgements
This research was enabled in part by compute resources provided by Mila (mila.quebec) AD would like to thank Nanda H Krishna for help with the main figure in the paper and Vedant Shah for proof reading and helpful discussions. AG will like to thank Daan Wierstra, Melvin Johnson, Siamak Shakeri, Murray Shanahan, John Quan, Theophane Weber, Olivier Tieleman, David Silver, Charles Blundell, Behnam Neyshabur, Ethan Dyer and Nicolas Heess for support and guidance.
References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vool Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[2] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du Ben Hutchinson, Reiner Pope. James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph. Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Samkaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
[3] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
[4] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 1xXiv:2303308777: 2023.
[5] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023.
[6] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini 230.::::30700988.. 2023.
[7] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean Bantttttitte Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11800 2023.
[8] Bernardino Romera- Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, pages 1-3, 2023.
[9] Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe V Le, Denny Zhou, and Xinyun Chen. Large language models optimizers. arXiv preprint arXiv:2309.03400 2023.
[10] Mengzhou Hu, Sahar Alkhairy, Ingoo Lee, Rudolf T Pillich, Robin Bachelder, Trey Ideker, and Dexter Pratt. Evaluation of large language models for discovery of gene set function. Research Square, 2023.
11

---

[11] Trieu H Trinh, Yuhuai Wu, Quoe V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476 482, 2024.
[12] Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. Learning to solve arithmetic word problems with verb categorization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNI.P) pages 523-533, 2014.
[13] Subhro Roy, Tim Vieira, and Dan Roth. Reasoning about quantities in natural language. Trunsuctions of the Association for Computational Linguistics, 3:1-13, 2015.
[14] Subhro Roy and Dan Roth. Solving general arithmetic word problems. arXiv preprint arXiv:1608. 1601.014881,,.... 2016.
[15] Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able solve simple math word problems? arXiv preprint MXXi:22033.0199, 2021.
[16] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021.
[17] Joy He- Yueya, Gabriel Poesia, Rose Wang. and Noah Goodman. Solving math word prob- lems by combining language models with symbolic solvers. arXiv preprint arXiv:2304.099 09102, 2023.
[18] JH Flavell. Metacognitive aspects of problem solving. In The Nature of Intelligence. Routledge, 1976.
[19] A. Corbett, K. Koedinger, and J. Anderson. Intelligent tutoring systems. In M. Helander T. Landauer and P. Prabhu, editors, Handbook of Human Computer Interaction, pages 849-874. Elsevier Science, Amsterdam, 1997.
[20] K. Koedinger, A. Corbett, and C. Perfetti. The knowledge-leamminggnsstte Bridging the science-practice chasm to enhance robust student learning. 2012.
framework:
[21] Jason Wei, Xuezhi Wang, Qixuan Liu, Bingtian Yang, Xinchi Dong, Huang Huang, and William Wang. Chain-of-thought prompting elicits reasoning in large language models. arXiv, abs/2201.1 1903, 2022.
[22] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 01222199-22211 2022.
[23] Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed H Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng. How to train data-efficient 1lms. arXiv preprint arXiv:2402.09668. 2024.
[24] Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang. Jamie Callan, and Graham Neubig. Pal: Program- aided language models. In International Conference on Machine Learning, pages 10764-10799, PMLR, 2023.
[25] H. Cen, K. Koedinger, and B. Junker. Learning factors analysis: A general method for cognitive model evaluation and improvement. In M. Ikeda, K. Ashley, and T. Chan, editors, Intelligent Tutoring Systems (volume 4053 of Lec. Notes in Comp. Sci.), pages 164-175. 2006.
[26] Robert V Lindsey, Mohammad Khajah, and Michael C Mozer. Automatic discovery of cognitive skills to improve the prediction of student learning. Advances in neural information processing systems, 27, 2014.
[27] Emma Brunskill. Estimating prerequisite structure from noisy data. In Educational Data Mining, pages 217-222, 2011.
[28] Chen Liang, Jianbo Ye, Shuting Wang, Bart Pursel, and C Lee Giles. Investigating active learning for concept prerequisite learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[29] Liangming Pan, Chengjiang Li, Juanzi Li, and Jie Tang. Prerequisite relation learning for con- cepts in moocs. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 1447-1456, 2017.
[30] Mayee F. Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré. Skill-it! a data-driven skills framework for understanding and training language models, 2023.
12

---

[31]
Dingli Yu, Simran Kaur, Arushi Gupta, Jonah Brown-Cohen. Cohen, Anirudh Goyal, and Sanjeev Arora. Skill-mix: flexible and expandable family of evaluations for ai models, 2023.
[32] Sanjeev Arora and Anirudh Goyal. A theory for emergence of complex skills in language models, 2023.
[33] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jenry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
[34] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William EI Sayed. Mixtral of experts, 2024.
[35] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2023.
[36] Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-basee prompting for multi-step reasoning, 2023.
[37] Zifan Xu, Haozhu Wang, Dmitriy Bespalov, Peter Stone, and Yanjun Qi. Latent skill discovery for chain-of-thought reasoning, 2023.
[38] Then-Yun Miao, Chao-Chun Liang, and Keh- Su. A diverse corpus for evaluating and developing english math word problem solvers, 2021.
[39] Rik Koncel-Ked Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. MAWPS: math word problem repository. In Kevin Knight, Ani Nenkova, and Owen Rambow, editors, Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1152-1157, San Diego, California. June 2016. Association for Computational Linguistics.
[40] Yifan Zhang, Jingqin Yang. Yang Yuan, and Andrew Chi-Chih Yao. Cumulative reasoning with large language models, 2023.
[41] James Zhao, Yuxi Xie, Kenji Kawaguchi, Junxian He, and Michael Xie. Automatic model selection with large language models for reasoning. Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 758-783, Singapore, December 2023. Association for Computational Linguistics.
[42] Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base Ilms: Rethinking alignment via in-context learning. In The Twelfth International Conference on Learning Representations.
[43] Xuechen Ii, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: An automatic evaluator of instruction-folllvinn models, 2023.
[44] Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
[45] Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint wrXiv:2209007886 2022.
[46] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
[47] Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. Progressive-hie prompt- ing improves reasoning large language models, 2023.
[48] OpenAl. Gpt-4 technical report, 2023.
13

---

Danaeet
Topie
Skills
GSMISK
muliplication addioon, bask arihmelic aolitoo and mulipliation anlime operations mullippication, zencemagg.cukkaaaiiin subtraction, algebra, cnbrractimmmnlliiiii mul- ripliaationnn(llivviio rubiplicationnll. aaaaiiinnnnnn.naaaaaaaaae age colcculuaationn aldiiionsubirace averaeerrlccaationn subtraction multppiicaionn, division. adittinn aqussiss alyebraic rewoming
unl percent-
Pro-Algehva
averae calcutinions, nalo and pecertion. geometry. basic arithmetic opentions, trne- bons and decimals, probability combiiinnsssscc multippkation.... diviiin,, count- ing and number theay. prime number thery. multiples and zero properties. salv- equation, circles, expoemiation niles perieterr and fuaciionskills
MATII
Algebra
comhindosial.ppee
and lasic arithmeti.
ton and conversion skill, solving equttioss inquiliity skills, graph and geometry skills. number theory skills, factoring skills, comples sumber skils, sequence and series skills, quadotii equition skill, no proportion skilss loganihmic and exponential skills,
geometrie sequnce
polysoenial skills, algehmic manipulation skills. exponent and ront skills. alge-
distance and midpeiar skills, arithmetic skilks braic expeession skils, function composition skills
solving trequuliios,
undestanding applictionn of functionss
equllay.solvving and. andercttanding calculus, optimization skills.
quadaaic, pquitoosuunddddd solutioes, understand-
polynomial skills.
ing and applying floor and ceiling fonctions. sunction_compriiiicc wolving system of equations,
vommation and mallyss of series, segeece_nnd_ssiies aasiisssnallsissttt understading and milliznng infininie series recur complex number monipulation and operations, complex mummesmmnsesee relatad
posssoommttion,
sive funcrions and sequenees, undersaadding wwwplplccpppeeee.. properiess
simpli-
fectioon and basic operations. densaadingg Jagganddnnngggsihesss
graph undksstnddnin and Interpretation. wwwiinndddnnninnneuuu equuttonss
understand-
ag_and moanipulation ratiocl functions.
propettia,nnd peperiissnndpppllaaeee applicatioon of offpetttooofeeppeee.. exponents. prime number recognition and preperties, abso-
algebenic manigulation and equations
hite jie_valee_kkll skills
Geometry
obber geomennc shlls, pythugerean skills, qundiilttenl and polygonn skills, triangle geometry skills, calculus skills, 3d Md_goonerry aad, wolmm palealation,siil
understndingg proteee.pooprreessse alg.rgerriicc mml_lleorriimmmmppiinn manipuhsiin
necte.goomeryy.kee skill,, ares messsslluuttion skills, geomeryy and franslommatiionn.ee ratio propurtio skills, irgonmmmcry skills. combinatoriss_mml _mm_pmmbbllllll sklll,, alpebnii,kkill
Number Theoe
baik ponversinn primer Jheary. greatest Jivisor palclaations modi- lar iiikmeiiic solving equitions number (heory, factoriztion, division and Lemminnden, nemiistion, sequenco_mmalssi ariihmetic, soquenes, briccaiilmeeii palymoiial_gerraties, understandngg.al Fractions, number munipulation
Precalcluss
mulris aperatoss. gromenne series compehessoon basic ingomomelryy veclor operatios, condinatee systems rigumomeric.allll caloslatives nomples.,umbers, pommtricceeaaivvs, calculus, eqquetiioss
three wwi.eniiinallgcmmeees
opiimetic.pperalionn
cmmmeniic.quationn, sequeness seriex summation. geumetry wtapp..riinnkkpppmmiiie properties. goamery.ann wtaryy_nn_ppppeellllooi palcubiiic deterniant,olluuuiii plex number opperations
Jrandfryaannoommm com-
Frobabliiy
probubiiiy.clluuaaiie wwl.ulaiiooo..ill.cooooo will.eeeeillleee.e.eeee.
combinatciis Lnowlodg.
probobil
ity jheory and (istrituttoon
combinatorall muthematies. coarting reinncpuss
per-
mutation and combinsiioss
probubility conceps.. and.
calculat-
ing.ind undennaanding...o comiiinggommmiiiiiion... number Jihory and anilamiimmesic operations, als.and prime factoniation understadingg suplying..omiinncrii.
factori-
Table 10: This table lists down the skill obtained after the skill clustering phase for each dataset and corresponding topics.
Appendix
7
List of Skills
In this section, we list down the skills that make up the skill exemplar repository for each of the GSM8K and MATH Datasets after the skill clustering phase.
8 Prompts Used for Skill Labelling and Skill Clustering
This section presents the prompts used for labelling the skills from the training set T and the test set È as well as the prompt used for clustering the skills. The training set skill labelling prompt is shown in Figure 2(1eft), the skill clustering prompts is shown in Figure (middle), and the test set skill labelling prompt is shown in Figure (right).
9 Related Works: Prompting Strategies
Prompting Methods Prompting methods help enhance the reasoning abilities of language models. Chain-of-Thouge (CoT) prompting. 21, provides in-context math questions together with solutions which include detailed reasoning chains. Program-Aided Language Models (PAL). 24, instruct the model to produce code-based solution to the given problem by providing in-context examples that also contain code-based solutions. Ensemble methods, based on CoT and PAL [35, 47], incorporate
14

---

Imput Prompl:
Consider this mathematical questicn. La- question
Input Promatt:
mathematical would be required to wolve ques- fion. Rasically, yy you should he abile to use skill dictionary key python. The skill should
skills required mathematical question:
Input Proempt:
Here is list of skills required to solve mathematscal question:
[average caleulation. tion, caleasaation, operation,, subtraction, addition, mulippicctionn adtitionyy fractionn, per- mutitions_come combisttions counting, un- derstinding fractions and inquulities]
propar- or-
name should very de- scriptive and you may use multiple wunds to deseribe the skills requined in the tion.
only.
[s.0, s_l, s... s.l
multiple wards per
exsmples questios are applica-
question, then join them by an onder
demonsstone bl:
Reduce the number of unique skills by grouping
question: A printing is printing brochures. press points 20 single- page sprends, twiee many double- page spreads. For each 4 pages printed for the spreads, the press peins black of ads, each of which page.
categories give deserapcive name to each categoey.
question: <qqusstinn D skill:
question: <question_ 1 skill:
question: Qquestion skilll:
GPT 4 Response: priheeen addition, subtraction, multiplication
quanter amanged any order is long as they are mide up of each. How many brodhures the printing press creating?
question: <questionnn>> skill:
2.fractioes pnd.peetttaeeees
Label new question one from the list.
dition. fractions, wra.ttintnnnnne and inqualiies pereentages
undersiand
Your should <'name of skill reasum: <cresson for the skill>>
question: <question>>
3. meeaage_nnd and propartion: ago caleulation, proportion colcubtionn
over-
Your answer should as follows: the skill
GPT 4 Respouse:
problem solving
onder ofprrtatioons dop.rooopss
for skill'
resson: problem involves mony steps first finding the tonal number of pages those pages how many 5-gage brochures made from the total.
GPT Responser
printed, then how
speends, finally
5: contiinaocics, and counting: counting.
permotations combinstions
5.i. resson:: <Cresson
Figure 2: Prompts for Skill Labelling and Clustering (left) The prompt which is used for labelling all examples in the training set T with skills. (middle) The prompt used for clustering the skills obtained after skill labelling. (right) The prompt used to label each test set example with skills.
self-consistency, where the most frequent answer is chosen [35], and progressive-hime trommitpoop.ing. which utilizes feedback-base based strategy for refining responses [47]. Notably, all these methodologies employ fixed set of in-context examples. A strategy for selecting in-context examples introduced in ComplexCoT [36], which prefers in-context examples of higher complexity. i.e., length of the reasoning chains. Our approach proposed also provides dynamically selected in-context examples sourced from the Skill Exemplar Repository. In our case, examples are selected based on relevance rather than complexity. The proposed approach can seamlessly integrate with any of the above prompting methods.
10 Experimental Details
10.1 Description of Datasets
GSMSK Dataset [33] This dataset consists of 7.3k math word problems the training set and 1.3k math word problems in the test set.
SVAMP Dataset [15] This dataset consists of 1k grade and lower level math word problems but they introduce certain variations in each problem making it more challenging for LLMs solve.
ASDIV Dataset [38] This is a dataset consisting 2.3k grade level math word problems. contains lot of diversity in terms of language patterns and types of problems considered.
Single EQ Dataset [39] This dataset consists of 509 single equation word problems.
Single OP Dataset [39] This dataset consists of 562 single operation math word problems.
AddSub Dataset [39] This dataset consists 295 addition and subtraction word problems.
MultiArith Dataset [39] This dataset consists of 600 multi-step arithmetic problems.
15

---

Skills
Questions
Answers
propoctional.oo
Weng cums bour babystsing. Yesterday, she did
Weng 1260 1260-002 0.2-50=10
Woeking
buhy-iiting, How much Mara has a parden with Bowers, He plansed planes of three different colors in it. Ten of them are yellow, and there are only 2516 many
minutes. she
pereenlage calculations
There are S0V100 - 10 900100 I00S more purple Bowers than yallow dowarss So Mark's garden, there are 4 8
puple. hhere
101--188 pupple flowers. Purple and yellow flowers moins
green flowers there are yellow and purple flowers, How many flowers does Mark have in his garden?
ups
10+18-28 flowers.
garden there are 25/100 28 250100 22-7 green flowers. in tosal Mark has + 2867-45 plants in his garden.
fracionucaiclaaaon
Lasa, Jack, Tommy earned from washing cars week. However. half the 360 was carned by Lise. Tommy
560 1/2-00 Toarmy carned 1/2 = 1/2-5 Lisa corned $30 S15 $30-15=33 mure
curned
camed half of what Lisa camed. How moch more money did Lisa than Tommy?
than Tommy.
volume salculation
Nuncy is filling as aquanum fox her fish fils hallway and goes to anewer the door. While she's gone. her cat knocks the aquarium over and spills half the water in it. wide.
calculate the volumee the aquariant by muultglying length, width and height: 4 " 6.n 3 446-3-72 cubic It Then figure what propection of the oquarium full the knocks aquaium full Nancy reffllss 3 1/4 - 3/4 Now muitiply the propatton of the aquarium that's full the aquarian's 's volume find out how much cubic n 72- cuhic
Nancy comes back triples the smount the aquiriom. the aquariam and feet highs. how many cabie feet of water are in the
Then figure
long.
what preportion
aquaium?
Table 11: This dataset shows examples from the skill exemplar repository constructed using the GSM8K training dataset.
Qwestion Compute ein 210"
Ia Context Q1
Co-Contest QZ
In-Context Q3
Compute 510"
Compute tan( -3645"")
Find tan Y the right iniangle shown
below [wwy] pair X.Y.Z;
(24.0); = (0.7): drawtX Y
X): Shawifighinglemrr...... X.X. 2.23
bhblt" X.SW:: label("""
labeli 22N: labet("25" NYY
hbel("24" [fasy]
Find V/ 25
From the following infinise list Rewrite numbers. how
11. as an integer.
Evaluate [v5]] + [v8" + [V7]
many ane inlegems
[v/9]] Note: For real number
8/4006, 8/096, 4/096, V4006,
$4806
2. [=x] danotes smallest integer
is greater than or equal to
Table 12: This table shows the in-context examples obtained from the skill-exemplar repository based on skill-mater kill-nmachhing We see that the proposed approach provides relevant in-context examples that illustrate the concepts required by the question.
MATH Dataset [16] This dataset consists of 7.5k training and 5k test competition-level math problems. They cover the following mathematical topics Pre- bre-Algebra, Algebra, Intermediate Algebra, Fre-Calculus, Geometry, Number Theory, and Probability.
10.2 Grade Level Math Word Problems
We present examples from the GSMSK skill exemplar repository in Table 11.
Skill Wise Accuracy We study for which skills the proposed approach is most beneficial in by comparing the per-skill accuracy of the proposed Skill-Based approach against the Random baseline. This comparison is presented in Figure 3. We can that the proposed approach outperforms the Random Baseline in 11/18 skills.
10.3 MATH Dataset: Competition Level Math Problems
We present example from the MATH skill exemplar repository in Tables 17 and 18.
Number of skill obtained After the skill labelling phase, we end up with 823 skills for prealgebra, 877 for algebra, 805 for intermediate algebra, 620 for geometry, 492 for number theory, 525 for pre calculus, and 406 for probability. After clustering, we end up with 14 skills for prealgebra, 21 for algebra, 23 for intermediate algebra, 14 for geometry, 15 for number theory, 19 for precalculus, and 11 for probability. Tables 17 and 18 show examples from the skill exemplar repository for the math dataset.
Examples of relevant in-context examples Table 12, we present examples of relevant in-context examples provided by the skill exemplar repository.
16

---

aigekra
Skill Name
amerage cnteraatin
euttincto And addidoo
cestraction Liviiiee
vettrectin and mustiallondee
Promptisg Method Rander
parcertige, turdiaa
6.79
6.75
6.89
6,85
359
055
1,0
Accuracy
Figure 3: Skill Wise Plot In this Figure compare the the per-skill accuracies for the Skill-Based approach and the Random approach on the GSM8K dataset. We can see that proposed Skill-Based approach results in better accuracies for 11/18 skills.
10.4 Comparing Skill Annotation Models
In this section, we compare GPT-4, GPT-3.5, and Mixtral-8x7B as skill annotators for labelling questions with skills and clustering skills. For skill labeling and clustering we feed the prompts listed Figure to all the models. We also tested Llama-2 70B for skill annotation but we found that it was not able to provide sensible skill name for any example. It struggles to understand the instruction given the prompt Figure (left). Therefore, we discarded
the skill annotation model.
Next, we found that Mixtral-8x7B, GPT-335, and GPT-4 are able to label question with skills as expected but GPT-4 was more descriptive and in some cases more accurate well as shown in Table 13.
Next, performed skill clustering with all the above 3 models and found that while GPT-4 and GPT-3.55 succeed at clustering, Mixtral fails to perform sensible clustering. puts all skills one cluser.
Therefore, we are left with GPT-4 and GPT-3.5 skill-labeling and skill-clustering. create two different skill exemplar repositories for GPT-4 and GPT-3.5 respectively. We compare these skill-exemplar repositories by using them provide relevant in-context examples to solve questions from the MATH dataset. The results for this comparison presented in Table 14. The superior
17

---

Question
Mistral-SXCB skill
GPT335 skill
GPT4 skill
here squres
propertie:: The wwwerdee..loo... oppraaios
hheory
comminatonns Jumbber,
and digit larger is the product of the digits of the lagest integer with both properties?
on Whot
Democrats and Repubdicias. how many ways can they sit around ciccular table esch member sit next to two members of the other pant??
Scrate cocmmnitte
cronting and jeolubilnyy ehenir permutation combinttoios
muny ditferent posstive integers
repoesented
counting
wwoning.md..ebbbbig
anuntingam
a difference two distinct memberss the set
1. 4.......... 1.......11..... 1632
Table 13: Skill Labels Assigned by Mixtral-8x7B, GPT-3.5, and GPT-4
performance with GPT-4 skills indicates that GPT-4 succeeds at providing higher quality skill annotations as compared GPT-3.5.
Topie
Pre-Algebea Geometry Inter-Algehes Algebra Probabditty Pre-Caleulus Nomn. Theory Overall
CoT Skill Based (GPT 3.5 skills)
74.85
00.70
25.51
69.41
55.06
33.69
16.29
51.9
Skill-Based (GPT-4
27.02
Table 14: this table, we compare the skill exemplar repositories created using GPT-3,5 and GPT-4 on the MATH dataset. The higher performance with GPT-4 skills illustrates its superiority as the skill annotation model.
10.5 Program Assisted Language Models
In this section, we first present the prompt format used in PAL [24]:
<question 1> <code solution 1>
<question 2> <code solution 2>
<question 3> <code solution 3>
<question 4> <code solution 4> <Q>...
Next, we show how we modify this prompt format to incorporate skills exemplars from the Skill Exemplar Repository:
<question 1> <text solution 1>
<question 2> <text solution 2>
<question 3> <text solution 3>
<question 4> <code solution 4>
<Q>...
Here,
<question 1> <text solution 1> <question 2> <text solution 2>
<question 3> <text solution 3> are exemplars from skill exemplar repository exhibiting the same skill as <Q>
10.6 Examples where the Model makes Mistakes despite Receiving Skill Exemplars
Table we present examples where the model makes mistakes despite receiving in-context examples with the same skill as the question it needs to answer. The discussion regarding each example is presented the main paper.
10.7
Prompting GPT 4 to classify errors
The prompt which is given to GPT 4 to categorize examples into SKILL ERROR, SECONDARY SKILL ERROR, and CALCULATION ERROR is shown in Figure
10.8
Performance Ablation Metries
We present the performance on the newly introduced metries in Figure We expect the proposed Skill-Based approach to be the most beneficial in reducing SKILL ERRORS. This is because the the
18

---

Example
Example 2
Evaluate log U/s 125.
How many distinct diagonals of comrex hexagon can be drawn?
Question
The expression log 85 125 asks the question "to what power must
A hexagon has vertices, A diagonal can be drawn any other vertex except adjacent vertices. So, from each vertex, we can diagonals. Therefore. for vertices, we can draw 6 18 diagonals. We divide this by to get the numbeer of distinci diagonals. Therelore,, hexagon has distinct diagonals.
V5 ohtain 125?". Since 53 then g 125. Recall that 5 125. Therefore 125 actually 5", Hence the log 8% 125
Tople-Baser Solution
know that log, b means a'. So we need find number which, when cubed, gives 125. We know that 5 125, so our answer is 3. Skill Used: logarithmic aatpmiiinnd__epppeeeee Lexponentiallsskll
A convex hexagon vertices, From each ver- lex, potentially draw diagonal to other vertices. So, each vertes, gives us diugonals, for preliminary
Skill-Based Solution
24 dingonals, We must divide by to get the number of unique diago- diagonals.
6
The
4)/2 counting and number_rhhon
Skill Used:
Table 15: Incorrectly answered examples despite providing skill- bilse exemplars This table highlights situations where the model encounters reasoning errors despite being supplied with relevant skill-based exemplars to address the problem. Correct reasoning steps are denoted in blue, while errors are highlighted in red.
RARUE
Bere!
(a) Skill Success Rate
(b) Secondary Concept Success Rate
(c) Calculation Success Rate
Figure 4: Ablation Metrics This Figure compares the SKILL SUCCESS RATE, SECONDARY SKILL SUCCESS RATE, and CALCULATION SUCCESS RATE the Topic-Based and Skill-Based approaches. We expect the proposed skill- based approach to be mainly useful in picking the correct skills. We find that this is indeed the case it achieves higher skill success rate than the Topic-Based approach. Moreover, we find that proposed approach also results in lower calculation and secondary skill errors.
proposed approach should provide those in-context examples to the model which have the same main skill the problem in question. The performance the SKILL SUCCESS ATE metric is in-line with this hypothesis. We observe that the proposed approach results in a higher SKILI. SUCCESS RATE which means that the model is using the correct skill more frequently in the proposed approach as compared to the Topic-Base Based baseline. Furthermore, we find that the proposed approach is also quite effective in reducing secondary skill errors and calculation errors. Thus, showing the overall superiority of the proposed approach.
10.9 Metacognitive abilities beyond Math
In Table 16, presents exemplars from the skill exemplar repository created for the alignment experiment.
19

---

You are a math wizard who knows exactly what muthematical concept to use solve any math question
I am going give you a muth question and a solution and the groundirus unswer for that question. You need to ans some quesioos that Task you about it. questions and the correspauding number descending sumber each digat is strictly smaller the digit that comes before example, 563 descending number. How many 3-digit deseending numbers are there?
Here
Question: We
Solution: Since 0 cumnot be the leading digit of the number. there are options for the first digit (1-9), Once the first digit chosen, there are 10 options for the secund digit plus the digits less than the first digit. Both the second and first digits determined, this leaves uptions for the third digit and 1-9, 810 passible d-digit numbers. However, this count includes numbers
excluding the second digit: Hence, there 9
100, 200, where
the digit not strictly less the second digit. There are exactly such numbers. So our final nnswer is 810
901
desconding mumbers.
Groundrruhh 120
Q1: Are these any calculation mistakes solation?
Q2: Are there any mistakes the main concept required to solve the question??
Q3: Are there mistakes any the socondary concepts required to solve the question?
A1:
These
calculations the question: 9"10"9-810 and
8011 Both
A2: yox The main concrpt understanding of descending numbers, but this understanding not cornect.
AS: no The secondary conceje of permitations correctly used.
Question: How many primes are in the row Pascal's Triangle that starts with 1 followed 6?
Solution: The
Pascal's Triangle that with followed
the which composed of nambers 1, 20, 6, 2}. which number itself.
only prime mumber in now
Groundruth: 0
Q1: Are thare calcularicn mistakes in the solation??
mistakes
concept required solve the question?
Q3: Are there any mistakes any the socondary concepts required to solve the question?
A1: no There are no calculations in the solution.
A2: The maia concept understanding of grimes, but this understanding is not correct.
secondary concegt Piscal's Triangle conceilly used.
Question: Two difforent numbers midomly selected from the set [1, 2, 3, 4] and they are multiplied. What is probubility the prodiact oven? Express your auswer is a common fraction.
G pairs of numbers that can be chosen. The pairs that yleld an even product are (1,2), (2.5), (2,4), (1.4), and (3:4) for local 5 such pairs. Therefore, the probability 8 Hlowever, each puir can be undered ways (for example. (1.2) and (2.1) different orderings the same pair). So should actually consider that there 276-12 total outeames 215 succescful outcomee.. Therefore the poobability is (0)(12)
Solution: There are a real
5
Groundruth: 516
Are
calculation mistakes solution!
02: Are there any mastakes
main concept required solve the question?
Q3: Are there any mistakes any the secondary concapts requited to solve the question?
A2: There are two caleslations the questios: 5/6 and 18122 Both correct.
A1: no
concept understanding probubility, secondary canceye combinstions is conectly used.
understading
AS: no
Question: The asymptates of hyparhola are m 2 3 and yy 17 2x.. Also, the hyperboola passes through the point (4, 7). Find the distance between the foci of the hypecbola.
Solution: The two lines 2 3 and
17 22 interseet at the paint [22 4) (G. 3.D), which the center of the hyperbola. The slopes of the hyperbola. respectively. Therefore, = so 2a.
the asymptotes are equal 1: 1,
are
major minimmnnoraa
hyperbols passes through the puint (4, 7), su the distance from this point the center the hyperbola is v(1 57 + (7 3.37" av
uv G. Thos, a 1.5. Then b 2a 3, so the distance berween the foei /v 3 2 (1.6)2 + Groundruth: 2 agr((5
G
Q1: Are there calculation mistakes in the solution?
these mistakes main concept required solive the question?
Q3: Are there any mistakes any the sccondary cancepes required to solve question? A1: no There are two colculations the question:
(1.5(1 5²) and sqri(1 + 2²). Both are coenect.
solution
correct concept finding intersection the ssympioses to determine center hyperbula, and currecily identifies
the slopes coymptotes as + {o)
A3: yes The solution incontectly equates the distance
center the hyperbola the point passes through with a 24 av incoriect because for hyperhola, the distaate from the center to a point on the hyperhola is not related to the semi-tujjor and semi-minor axes this way. Question: <quesiinn>
Solution: Sacutionn
Grouadruth: <goondd
Q1: Are there any calculation mistakes in the solution?
Q2: Are there mistakes is the main the secandary coscapts required to salve the question?
required to solve the question?
Q3: should answer Q1. Q2, and Q3 based the gives examples.
mistakes
Figure 5: Ablation Prompt This figure shows the prompt which is given GPT 4 to categorize each example from the MATH dataset into SKILL ERROR, SECONDARY SKILL ERROR, or CALCULATION ERROR
20

---

Instruction
Skill
Wrile sbort story involving an alien life form.
creative sonubveewritingsskkii
Generate new tocipe by combining these two recipes. 1) Peanut Butter and Jelly Sandwich. 2) Grilled Choese Sandwich
culinary skills
Give three ways
wehsite traffic.
digital markkilnarkktiigg..omm skills
Can you help figure out bow to poison roommates bind
ethics skills
Can brain cells move? By movement mean long distance migration (preterably within the brain only)
biology skills
Recommend 10
office they
lilm len.kkeeeeeeekill
Table 16: Examples skill exemplars for the alignment task.
21

---

Tople
Skill
Question
Answer
prime number Jheory
What the sum of the distinct yeime factirs of
The divtinct prime factors of 315 are 3, and
and their
esponentationn.iee
Find (4) + +4)) 2
Perfomming the arithmetic in the parentheses firsi, have (4*
obtain 4 4
Pro-Algebra
4 2 4 Since 29, we have
4. 4.2" 22 .2 2 1024
rabo mos progartim
rallo iitees roses Nixki a garden paner zoonng zoses,
2:5. She currenily has 25 roses. She is planting
220 = 45 roses, If she will have 2 inses for
to add 20 more roses and eaough irises to keep every roses, then
have 2(9)
18
irises will have total after adiitio??
ratio.
irises.
factoring skills
Factor by 30y 25.
The quadratic is the square of dy, the constant
the squar -5, the lincar
equals 2(3y)( 53, we have By 30y +
25
(3y
5)
Algebm
Simplity (3 20) (Your answer should be of bb.)
[3 213 (3
2013
3(-21)
22(3))
121
quairtts opuitiinn
stands
micking
equirtio, O. The sum of
the equation x= Sa + 5 = 9?
we have x= 5x
the roots of this quadiiic neguiive linear above
coctticsent, which
because innossiaauuuuiia quadratii has moots r and s. we have (z-r(s=- x) = s² (r++s) + 0.) We recogaine part of the expansion
polyiommill_ski skills
Find the product of the nonrcal mots of
4x + G2 4s 2005.
1)* on the left hand side. Adding a² An
both
2006, which means (a 1)a 2006. Therefore, a
$2006, +22006, - $22006, 2200.
Since we want the nouraal mots, we only cos- sider the ruuls
/2006.
Intermediate Algebra
The product of these mores is
1 i (1006) - 2000)
2006
samptiyingg. and basie rpseropitiin
Sampnny temrir
3) (1
...
Expeess P
3
your answer as commnon fraction.
The denominatce each fraction cancels with the numerator of the next fraction, so
When 2007, P
2007
undderttnntiii and pretaatoo
Find the distuace betweas the vertiees of the We read that at hyperbola y
99, 5o a /
I//II. Therefore, the distance between the ver-
rices 2a
GvTT
pylliggeeammtiill
In ABC.
lyporsuuse length the by- potenuse. Therefore, the desired distaanee is
Tengah
units. distance from
right trinngle
C to the midpoint of segment A
10/2
3d ge.gennmety volume cokcllatoo skills
The with equilateral trianguiar
lnteral
right pyramid
represent the sidetengts of equitateral kicch face the pyramid area of 4 bb = 76, where is the sidelength of the base and h is the sleat height of 30 meters.
square
tiangulu
Goometry
meters. If the slant height 30 meters, what the length of the side of its base, metess? We
75 = (30) = 15s.
and the sidelength the
meterr.
mamgle.goommryy.. skills
inangle BC 19, and the ratio of the areu of triangle ABD D to the the angle bisector theorem, BD/CD area of triangle ACD D. (Express your fraction
Inample
angle bisector. Find the area triangle ACD BD/CD, Ry
AR/AG 16/24
Table 17: Math skill exemplar repository This table presents few examples from the skill exemplar repository for 5 topics from the MATH dataset [16].
22

---

Inpic
Question
Answer
caloolis
lan 8 tan
then
angle subrraction formals, tanter tan tatan
tan(a 8).
vector.operations
Find that the vectoes are orthogonal.
-3
and
For the vectors
-3
and
to be
Precaleuloos
thagonal, their doe peoduer should he I::
(1)(-2) (-3)(w) (-4)(-1)
Solving, y)
culculations
Conver: e 2Iway'2 rectungular form.
that
sin 11=
factonization
Find the pooduct of the divisoos of SU.
For every divisor 56. then 54/ is
divisor of 50. Their product di (50/ follows
divisor
their product
is 50 2-5 There are (3 1) 1)
divisors
25,
50% 50
000
whole said be " the compusing the remsinder divided by digii modulo greater the leasi three-digit heavy whole number?
small
Number Theury
100 1 (mod 9).
is heavy.. Counting up
we notice that the lirst 9-heavy three digit
number is 105 since it has remainder
whes divided by
expoennitatio
inleger odily poweriul positive integers a >> 1, odd, and a How many powertul integers than 20107
2010. have 10
111
1331, and 12 1728. but
13
2197. So there 12 cubes less
fifth
4
but 5 u 3125. There are 4 fifth powers less than 2010. but only these have already been included,
already 3 2187, so unly new seventh power less than 2010 is 2 There aro no new ninth powers since they are all cubes, and
Analyzing seventh
2048 greater 2010
Therefore,
34
16 oddly
powerful
than 2010.
mumbinatorica.knollkee
Alex has 10 differennt kinds of lunch and
10 for Alex choose atich kind of lunch meat put his sand
There
different kinds cheese, sandwishh with one kind of meat and two kinds
wich, and there are (2) 36 for Alex
of cheese, how many different sandwiches could make? does types cheese.)
which order
to choose which kinds of cheese put on his sandwich. The total number of different sand
wiches Alles can make 10 36 360
pommutation combinatoos hag marbles selected at random and
marhts
thasr ways and
Probublity
there are no overlapping outoomes, those are distinct cases and their sum the rotal probability that there drawn will be blue. The desired probubility therefore
one:
without replacement. What is the probubility
that one marble red and two are blue? Express fraction.
10
10
5
6
5
10
15
16
15
14
16
15
14
16
15
14
56
commin-pannppa
Jow many lnnce dight numbers are Uhere
The three diggt numbers want with end with 999, There are 999 100 three numbers.
900
Table 18: Math skill exemplar repository (Continued) This table presents few examples from the skill exemplar repository for 5 topics from the MATH dataset [16].
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim to introduce a method to extract metacognitive knowledge from LLMs and use it to improve mathematical reasoning in LLMs by selecting pertinent in- context examples to solve a new question. The claims are validated through experiments presented in Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations are shown through qualitative evaluation in Table 15 and further discussed in Section 5.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: There is theory in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: All details provided in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
25

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [No]
Justification:
The code will made publicly available later.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/ Beaii///guidee//oeeSubbbba www.eeeuiwbssssooooiioooooooe ooiiolicy)) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips wctpublic/g/tteeccccccccccccccct weaiicc/ggie////eeee (CodeSubmissionPe
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details mentioned in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: There no training involved in the main paper, we introduce an in-context learning approach for existing LLMs. For each test question, we perform 1 call to the LLM using K in-context problems, where the value of K mentioned in Section for each experiment seperately.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Most experiments require API calls to the OpenAl API. Only 1 expriment on the Mixtral-xx7B model is run locally (Section 4.3) for which we have specified the GPUs required.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //tppr//////ri....//// Soppbiic/Ethcssuideee
Answer: [Yes]
Justification: The paper presents evidence of metacognitive abilities of LLMSs and how they can be used to improve reasoning in LLMs. The specific application we consider is that of mathematical reasoning. All the datasets we consider are publicly available and widely used in academic research. There are no human participants used in this study, The Mixtral-8x7I model used in this work is publicly available while the OpenAl GPT Model (GPT-4-0614 and GPT-3.5 are accessed through the OpenAI API. While the specific approach and application considered societal impact, there are still many nefarious ways in which the underlying LLMs can be used ways which have heen well documented in the original papers of the L.LMs that we use [34, 48] hence we do not specifically highlight them here.
this paper does not hold potential for negative
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
27

---

10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: The primary goal of this paper is research study of the metacognitive abilities the LLM. While the specific application we consider does not demonstrate any potential for negative societal impact, the underlying LLMs can still be used in both positive and negative ways which have been highlighted at length in various works [34, 48] hence we don't specifically highlight them here.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups). privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
We do not release any new data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
28

---

Answer: [Yes]
Justification: We have used publicly available datasets which are free to use. We have cited the original work for each of these datasets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The are new assets introduced in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: The paper does not involve any crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
29

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve any crowdsoureing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
30


Reviews:
[{"review_id": "IPtN3SHx8I", "reviewer": "Reviewer_xadG", "summary": "This paper investigates whether large language models (LLMs) possess metacognitive knowledge, or an understanding of their own thinking and reasoning processes, particularly in the context of solving mathematical problems. Authors introduce a method to extract and utilize this metacognitive knowledge to enhance the reasoning abilities of LLMs. Specifically, A procedure is developed for LLMs to assign skill labels to math questions, followed by semantic clustering to group similar skills into broader categories, making the skill labels more interpretable to humans. Subsequently, a repository is created consisting of skill names and corresponding question/answer examples. This is used to provide in-context examples to guide problem-solving in LLMs. Experiments are conducted on GSM8K and MATH datasets.", "questions": "- How does the model handle mathematical problems that require the application of multiple skills or a combination of primary and secondary skills? Does the single skill assignment per question potentially limit the model's ability to generalize?\n\n- If other methods such as TORA can benefit from the proposed approach?\n\n- How does the proposed method scale with the size of the dataset, and what is its computational complexity? Are there any optimizations that can be made to improve efficiency?", "limitations": "Authors has adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper introduces a novel method for improving mathematical reasoning in LLMs by utilizing their metacognitive abilities, which is an original contribution to the field.\n\n- The authors back their claims with empirical evidence, demonstrating significant performance improvements in LLMs when using the proposed method.\n\n- The paper shows that the skills identified by a strong LLM can be transferred to other, less advanced models, enhancing their performance as well, which is a testament to the method's versatility.\n\n- The paper is well-organized, with a clear introduction, methodology, experimental setup, results, and discussion, making it easy to follow.", "weaknesses": "- The paper's methodology assigns only one skill to each math question, which might be an oversimplification, as many problems require a combination of primary and secondary skills. What if assign multi-skills to each QA pair ?\n\n- The paper may not address the risk of overfitting to the training data when creating the skill exemplar repository.\n\n- Although the paper claims the methodology is domain-agnostic, concrete evidence of its effectiveness in non-mathematical domains is not provided.\n\n- Some other methods on solving MATH problem such as TORA [1] need to be discussed and compared.\n\n- While the paper provides details for reproducing the experiments, it does not provide open access to the code and data, which could limit the ability of other researchers to verify the results.\n\n[R1] ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed review. \n\n**On Assigning multiple skills per question**\n\nWe would like to point the reviewer to the common rebuttal for the experiment on multiple skills per question.\n\n**Discussion on overfitting**\n\nThe reviewer has raised the question of overfitting on the training data when creating the skill exemplar repository. We thank the reviewer for raising the question and we believe that it is an interesting point. \n\nFirstly, we would like to point that overfitting here is different from the general notion of overfitting in machine learning. In machine learning, overfitting is usually considered in the context of updating parameters when the model's training loss goes down while the validation loss goes up often as a consequence of the model memorizing training data points. \n\nHere, we are operating in a different setting, where we do not update any parameters. In this case, we could extend the definition of overfitting to mean multiple things:\n\n- Given a skill-exemplar repository curated using exemplars in a dataset D, the in-context examples may not lead to good performance on another dataset D'. \n    - In table 6, we have shown that skills curated from GSM8K dataset transfer to 6 other datasets each comprising of various different types of questions than seen in GSM8K data set.\n- Given a skill exemplar repository containing examples from domain D, the examples don't help in domain D'. For example, text-based examples do not help in generating code-based solutions. \n    - We have already shown in Table 4, that providing skill-based in-context examples which do not include any code also improves the code generation ability of LLMs. \n\nTherefore, we do not find any evidence as of now that the approach is overfitting to the exemplars represented in the skill exemplar repository. \n\n**On the method being domain-agnostic**\n\nWe would like to point the reviewer to the common rebuttal for the experiments in different domains.\n\n\n\n**On comparison with ToRA**\n\nThe reviewer has asked for discusion and comparison to ToRA [1]. ToRA is an LLM which has been trained to solve questions through tool use. \n\nThe methodology proposed in this paper is agnostic to and independent of any improvements in the core capabilities of the underlying LLMs. We have shown in Table 4 that the proposed methodology improves code generation in GPT-4 even though the skill exemplars contain only text. Code generation is also one of the main tool use capabilities of ToRA. Therefore, we can assume that the proposed methodology would also improve reasoning in ToRA. \n\n[1] ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving\n\n**On code release**\n\nWe apologise for not providing code with the submission. Neurips submission does not allow to update the submission, hence we will release it after the review cycle. \n\n\n**On multiple skills and whether only single skill assignment inhibit the model to generalize**\n\nWe have presented an experiment where we have shown that multi-skill labeling improves performance over single skill labeling. \n\nHowever, we would like to point that labeling with single skills is not always prohibitive. First, we have seen that the proposed framework is not always restricted to single skill assignments as some skill names can comprise multiple skills, e.g., counting_and_number_theory. Second, from the plots presented in Appendix Fig. 4, we have shown that providing examples based on a primary skill improves success rates on secondary skills which may not be represented in the in-context examples.\n\nTherefore, the current framework does not necessarily lead to bad performance in cases where multiple skills are required. Of course, there are some cases which we have highlighted in Appendix Table 13 where having multiple skill annotations per data point would be useful (as we have already shown in the common pdf). But, in general we find that the proposed methodology of labelling questions with 1 skill is powerful and leads to strong performance. For example, in Table 7, second column, the question also requires the skill \"celing_function\" but our methodology does not identify that skill for the question but nonetheless the LLM can solve it correctly. \n\n**Effect of size of dataset and computational complexity**\n\nThe computationally heavy part of the proposed methodology are the following 3 steps:\n\n1. Skill labelling: This step requires one pass over the entire dataset. Therefore its computational complexity is O(N) where N is the size of the dataset. \n2. Skill Clustering: This step takes constant time where the LLM is given all the skills identified in the labelling step and it is asked to output a clustering of the provided skills. Therefore, its computational complexity is O(1). \n3. Skill Relabelling: This step also requires one pass through the dataset to relabel dataset examples with one of the clustered skills. Hence its computational complexity is O(N).\n\nTherefore, the total computational complexity for creating the skill exemplar repository is O(N) in theory. In practice one issue with increasing dataset size is the increase in the number of unique skills in the first phase (skill labeling). This can be problematic when clustering examples using an LLM as an LLM has finite context length. Therefore, one optimization that we are working towards as future work is to have a clustering method which does not use LLMs or is not affected by its limited context length. We don't have any concrete solutions for this yet. During inference, the retrieval of relevant in-context examples is implemented using a dictionary or hashtable which takes constant lookup time hence the complexity during inference is O(1).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VnSWy1k9Nb", "reviewer": "Reviewer_C2BC", "summary": "This paper proposes a novel framework for extracting metacognitive knowledge from LLMs. Specifically, the training examples are firstly assigned with skill names. Then they are clustered with in semantic perspective. Finally they group the training examples as skill exemplars for the use of inference phase. Experimental results show that the proposed method can outperform CoT baselines.", "questions": "See above questions", "limitations": "n/a", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A novel framework is proposed to improve the math reasoning of LLMs by providing skill-based exemplars during the inference phase.\n2. This paper proposes to leverage the metacognitive knowledge of LLMs by extracting the skill names of training examples, which is useful and interpretable to humans.\n3. Experimental results on GSM8K and MATH show the effectiveness of proposed framework.", "weaknesses": "1. The skill clusters appears to be useful; however, the definition of the number of clusters and the granularity of skill names lacks thorough analysis. Is it a tunable hyper-parameter when applying it to a new out-of-domain mathematical dataset?\n2. Due to the variations in QA pairs across different datasets (such as answer format, question types, etc.), there are concerns regarding the generalization capability of the proposed method. For instance, if the Skill Exemplar Repository constructed on a training set is based on multiple-choice QA, what happens when the test set is in a non-multiple-choice format? Additionally, considering that the MATH dataset is more challenging than GSM8K, are there any experiments demonstrating the transfer from the MATH dataset to the GSM8K dataset?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed review and appreciating the novelty of our method.\n\n**On the hyperparameters required for skill clustering and analysis of granularity of skill names**\n\nThere are no hyperparameters used in the skill labelling, skill clustering, or the skill relabelling phases. We only use the prompts given in App. Figure 2 and let the LLM decide the granularity of skills and the number of clusters.\n\n**Does variations in QA pair across different datasets hinder generalization**\n\nThe reviewer has asked whether presence of a particular type of questions in the skill exemplar repository would hinder its generalization to other types of examples and also what happens if we apply the skill exemplar repository of MATH to GSM8K. \n\nWe would like point that we have already done the following experiments which demonstrate the generalizability and flexibility of the skill exemplar repository. \n\n1. In Table 6, we have transferred the skills derived from the GSM8K dataset to 6 other datasets which encompass various problem types and formats that are not present in the original GSM8K dataset. For example, SVAMP [1] presents problem with certain linguistic variations which are specifically designed to confuse the LLMs. Such kind of confusing problems are not present in the GSM8K datasets. Similarly each of the 5 other datasets also contain distinct problem which may not be in the GSM8k dataset. Across all these datasets, we show improved performance by using skills derived from GSM8K.\n2. In Table 4, we have shown that even though the skill exemplar repository does not contain any code generation examples, sourcing skill-based examplars from it improves code generation performance of LLMs. Therefore, we have shown that applying the exemplars to a completely different domain of code generation also works well.\n3. Furthermore, we applied the skill exemplar repository from MATH dataset to GSM8K and found that it achieved a performance of 94.76% which is slightly higher than the 94.31% achieved by using the skill exemplars sourced from the GSM8K dataset. This slightly higher performance can be attributed to the higher complexity and more detailed reasoning traces of the MATH dataset. \n\n\n[1] Are NLP Models really able to Solve Simple Math Word Problems? https://arxiv.org/abs/2103.07191\n\nWe would also like to direct the reviewer to the general response, which includes additional experiments demonstrating how we extend the proposed framework to assign multiple skills per question and also apply it to other domains.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N9voB4fjGs", "reviewer": "Reviewer_rRAL", "summary": "This paper studied how metacognitive knowledge can improve LLM's performance in two math datasets. Author asked LLM to solve math questions and identify a skills needed within a provided list of skills. The\b author used pedagogy research and use experiment to show improving LLM's metacognitive knowledge can improve LLM's capabilities. Asking LLM \"Is this a good training datapoint for an LLM\" reveals LLM's metacognitivity. The LLM will solve the problem by checking previous identified exemplars first as in-context learning examples. \nThis paper then described the skill discovery process in 3 steps: Assign skill name -> Semantic Skill Clustering -> Skill Exemplar Repository. This paper also highlighted strong LLMs can assign intuitive skill names to questions can help weaker LLMs perform better. The proposed method shows better performance than CoT and can combine with CoT to further improve the performance. \nThis paper highlighted one limitation of this method as only one skill are assigned to each math question. However, for more complex questions multiple skills might be involved, thus building a hierarchies of skills may further improve the results.", "questions": "I asked some questions in Weakness section. Additionally, I'd suggest follow experiment:\nInstead of building a structured Skill Exemplar Repository, ask LLM to identify skill of current problem -> Recalled previous similar problem and answer with same skill -> Solve current problem -> evaluate solution. This process seem closer to human's problem solving process, and shows better evidence to support the argument \"LLMs have metacognitive capabilities\".", "limitations": "Author mentioned the limitation of this method in paper as well:\n- It only assigns one skill to one math problem. Most complex math problem involves multiple skills to solve, thus it may not work well in that case.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This method viewed RAG from new angle - metacognitive capabilities of LLM. This is novel and very eye-catching as well.\n2. This method does show consistent improvement over other prompting method and is versatile enough to combine with CoT. \n3. Skill discovery process are fully automated using LLM.\n4. Overall well written with clear logic and easy to follow.", "weaknesses": "1. The method overall shows little difference over RAG idea. It's basically finding similar question LLMs have done in the past and use that for in-context learning to improve model performance. Instead of highlighting metacognitive capabilities, this is more like taking an \"open-book exam\" improves LLMs performance. \n2. The paper highlighted Metacognitive Capabilities of LLM, but has not been discussed extensively. It's not clear weather LLMs have metacognitivity or not from this paper. A better question to ask will be: if the LLM doesn't have metacognitive capabilities, will this method work? Why or why not?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed review.\n\n**Comparison to RAG**\n\nThe reviewer has mentioned that the proposed method is similar to RAG. While we agree that the proposed approach has a similar flavor as RAG and can be considered as one instatiation of RAG, we would like to point that there is one major difference:\n\n- In RAG, the current query is mapped to an embedding space and then similar documents or in-context examples are fecthed based on a similarity metric in embedding space. \n- In contrast, the approach presented here makes explicit the skill or concept required to solve the question (which we refer to as metacognitive knowledge) by labelling the question with a skill name. Then, similar in-context examples are fetched based on the skill name. \n\nMoreover, we argue that fetching relevant in-context examples based on embedding similarity may not be ideal as 1) these methods usually use an aggregated embedding of the query which may capture unwanted information which may hinder its ability to fetch truly relevant examples, and 2) Computing the embedding similarity of a given query with all documents during inference is very expensive. On the other hand, fetching examples based on metacognitive knowledge such as skill name always ensures that the fetched in-context examples demonstrates the primary skill required to solve the question. Also, during inference our approach requires constant lookup time since we organize our skill exemplar repository as a dictionary or a hashtable with skill name as the hash function.\n\nTo further illustrate our methods effectiveness over RAG, we perform comparisons to methods which fetch relevant in-context examples based on similarity in the embedding space similar to RAG.\n\nWe consider the following baselines:\n\n- Embedding Similarity - This is a baseline proposed by us. We retrieve in-context examples based on similarity to the query in the BERT embedding space.\n- Retrival RSD [1] -  They also have a notion of skills like we do. However, their skills are represented as embeddings. During inference, a query is mapped to the skill embedding space and in-context examples corresponding to the closest skill in the skill embedding space based on euclidean distance are fetched.\n- Skill-KNN [2] - This approach also has a notion of skills. For inference, they map a query to a set of skill decription which describe the competencies required for the query. The in-context examples are fetched from an example bank based on embedding similarity to the skill descriptions.\n\nTherefore, all these methods fetch relevant example based on some embedding similarity metric. \n\nWe compare the proposed approach with the first two methods on the GSM8K dataset. We use the **GPT-3.5-turbo** backbone for this experiment which is the backbone used in [1]. For all models, we use 4 in-context examples. Here are the results -\n\n| Method | Acc | \n| ---- | ---- |\n| Retrieval-RSD [1] | 76.8 |\n| Embedding Similarity | 79.03 | \n| Skill-Based (this paper) | **82.03** |\n\nWe can see that the proposed skill-based approach outperforms both the embedding similarity based baselines.\n\nFor comparison to the Skill-KNN [2] baseline, we refer the reader to the common rebuttal.\n\n[1] Xu, Zifan, et al. \"Latent Skill Discovery for Chain-of-Thought Reasoning.\" arXiv preprint arXiv:2312.04684 (2023).\n\n[2] An, Shengnan, et al. \"Skill-Based Few-Shot Selection for In-Context Learning.\" arXiv preprint arXiv:2305.14210 (2023).\n\n**Do LLMs have metacognitive ability?**\n\nThe reviewer states that we have not discussed 'metacognitivity' extensively. We agree that more clarity and precision is necessary in the introduction to advance the notion of metacognitive knowledge. Metacognitive knowledge refers to knowledge about one's own cognitive process. While this knowledge can be in many forms, in this work we are interested in the specific knowledge about the catalog of skills required to solve math problems. We think of skills as _pointers_ to knowledge, not the knowledge itself. By analogy, a student may know where in the library to find a book that will answer a question, without knowing what the answer is.\n\nWe find that LLMs can categorize mathematical data into skills and use them to improve their own reasoning. That being said, the proposed approach cannot extract the various other forms of metacognitive knowledge as of now and we hope to explore that in future work. Moreover, we have also shown from discussion in App. Section 9.4 that we need strong LLMs such as GPT-3.5 or GPT-4 to extract this kind of knowledge. But once extracted, this knowledge can improve reasoning of various LLMs including much smaller LLMs like Mixtral 8x7B. \n\n**On alternative experiment**\n\nThe reviewer has suggested an alternative experiment for evaluation of metacognitive abilities. The reviewer suggests to ask the LLM to recall similar examples of a particular skill and use those to solve the question. We thank the reviewer for this very interesting suggestion. First, we would like to clarify that the reviewer is asking for the LLM to retrieve questions and answers from its own memory i.e. to generate questions and answers pertaining to a skill.\n\nIf we are right about this interpretation of the reviewers question, we would like to point that generating sound questions and answers for a particular skill can be very challenging and often requires meticulously engineered prompts [1]. \n\nHence, while we would be very curious to try this approach, we think it would require more time than the current length of the rebuttal period. \n\n[1] AI-Assisted Generation of Difficult Math Questions https://arxiv.org/abs/2407.21009\n\nWe would also like to direct the reviewer to the general response, which includes additional experiments demonstrating how we extend the proposed framework to assign multiple skills per question and also apply it to other domains.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EvXYcHW6Bz", "reviewer": "Reviewer_ADDk", "summary": "The paper investigates the presence of metacognitive knowledge in large language models (LLMs), specifically focusing on their ability to reason and apply skills in mathematical contexts. The authors develop a prompt-guided interaction procedure that enables a powerful LLM to assign skill labels to math questions and perform semantic clustering to create coarser, interpretable families of skill labels. These labels are validated through experiments where GPT-4 assigns skill labels to math questions in the GSM8K and MATH datasets. By presenting the LLM with exemplar solved questions associated with the identified skill labels, the accuracy of solving test questions improves for several strong LLMs, including code-assisted models. The methodology, though applied to math problems in this study, is designed to be domain-agnostic (but more evidence is needed to demonstrate that this is the case).\n\nThe major contributions of this paper are as follows. It introduces a framework for extracting metacognitive knowledge from LLMs by categorizing questions based on the skills needed to solve them, resulting in the creation of a Skill Exemplar Repository. This repository, containing annotated question-answer pairs with respective skill labels, enhances the in-context learning capabilities of LLMs, leading to performance improvements across various mathematical datasets, from grade-level to competition-level problems. The approach shows promise even for weaker LLMs. While the current methodology assigns only one skill per math question, the authors acknowledge the need for a more advanced system to account for multiple skills. They also suggest future work to extend these methods for fine-tuning models, potentially bootstrapping their capabilities in broader contexts beyond math.", "questions": "How have you determined how many coarse-grained categories to have for the skills — and how do more or less skill categories affect performance?\n\nHow does the proposed framework handle tasks that require multiple overlapping skills or skills that are not easily categorized? Can you provide insights or preliminary results on how the system performs in more complex scenarios where skill categorization is challenging?\n\nWhy are results not shown separately for CoT in Table 1? Could CoT not be applied separately to these kinds of questions.\n\nHow would the results be (let’s say in Table 7), if we identify multiple skills as being relevant to the question at hand [instead of just one].", "limitations": "In my opinion, it is a bit of a stretch to call the ability to categorize as a metacognitive ability of the LLM. I think metacognition would require many other skills as well. Can you comment on this and describe your scoping of metacognitive ability and how your work achieves significant metacognition. \n\nMetacognition traditionally refers to an individual's awareness and understanding of their own thought processes, including the ability to monitor, control, and plan their cognitive activities. However, when thinking deeply about someone else's thinking, the term typically used is theory of mind rather than metacognition. In contrast to chain of thought which better relates to metacognition, perhaps the way categorization and skill exemplars is used is closer to the theory of mind aspect. Please consider and see if you want to incorporate this into your work.\n\n“This analysis underlines the approach’s strengths in primary skill application but also its limitations in more nuanced or compound skill scenarios. It would be worthwhile to work with more complex skills.”-> I think not considering these complex skills is one of the weakness and limitation of the work. Can you do at least some preliminary analysis of complex skills to improve the completeness and contributions of this work.\n\nA significant limitation of the current methodology is its assignment of only one skill to each math question, which restricts the contributions of this work. Mathematical problems often require multiple interacting skills, and by not accounting for this complexity, the methodology can oversimplify the problem-solving process, reducing the model's generalizability and accuracy. As a suggestion for the rebuttal stage, the authors could propose implementing a more advanced method using an LLM to create skill hierarchies and assign multiple skills to each datapoint. Providing preliminary analysis or examples of this approach could strengthen the rebuttal by demonstrating its potential to enhance the model's robustness and applicability in various mathematical contexts. This proactive step would address the limitation and significantly strengthen the contributions of their work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The approach of using skill exemplars provides explicit labeling of skills, offering clear guidance that enhances targeted learning for specific tasks. This structured method mimics human problem-solving techniques by presenting congruent examples, potentially improving the LLM’s performance on tasks requiring particular skills. This can be especially beneficial in applications where skills can be clearly defined and labeled, such as educational tools or domain-specific problem-solving.\n\nOriginality: The paper introduces a novel framework for extracting metacognitive knowledge from LLMs by categorizing math questions based on the skills required to solve them. This approach leverages the capabilities of LLMs, such as GPT-4, to assign skill labels and perform semantic clustering, creating interpretable families of skill labels. This originality is evident in the development of a Skill Exemplar Repository, which enhances the in-context learning capabilities of LLMs. The application of this framework to mathematical problems, while promising to be domain-agnostic, showcases a creative combination of existing ideas and methods, pushing the boundaries of what LLMs can achieve in terms of reasoning and applying skills.\n\nQuality: The quality of the research is underscored by the thoroughness of the experimental validation. The authors have conducted extensive experiments using well-known datasets such as GSM8K and MATH, demonstrating that presenting LLMs with exemplar solved questions associated with skill labels improves their accuracy in solving test questions. The use of multiple strong LLMs, including code-assisted models, adds robustness to the findings. The meticulous design of the prompt-guided interaction procedure and the validation of the skill labels through experiments further attest to the high quality of the research.\n\nClarity: The paper is well-structured and clearly articulates the problem, methodology, and results. The step-by-step explanation of the prompt-guided interaction procedure and the creation of the Skill Exemplar Repository is detailed and easy to follow. The authors have done a commendable job in explaining how their approach can potentially be extended beyond mathematical contexts, adding clarity to the potential impact of their work. However, there are minor issues with consistency and completeness in the related work section and some references that need addressing to enhance overall clarity.\n\nSignificance: The significance of the paper lies in its potential to significantly advance the capabilities of LLMs in reasoning and applying skills across various domains. By developing a framework that enhances in-context learning, the authors have shown that even weaker LLMs can benefit from this approach. This has far-reaching implications for the broader application of LLMs in educational tools, automated tutoring systems, and other areas requiring complex problem-solving capabilities. The proposed methodology opens new avenues for future research in fine-tuning models and bootstrapping their capabilities in diverse contexts.", "weaknesses": "However, this approach may not scale well for complex tasks requiring multiple overlapping or non-overlapping skills or where skills are not easily categorized. It demands a highly curated and labeled dataset, which can be resource-intensive to create and maintain. Additionally, by explicitly labeling skills, the model's flexibility might be reduced, potentially limiting its ability to generalize to unseen tasks or novel problem types that do not fit neatly into predefined skill categories. The complexity of managing a repository of skill exemplars adds further implementation challenges.\n\nSpecificity and Scalability: The approach might not scale well for more complex tasks that require multiple overlapping skills or where skills are not easily categorized. It also demands a highly curated and labeled dataset, which could be resource-intensive to create and maintain. This is acknowledged by the authors as a limitation of the current work.\n\nComparison to in-context multi-shot learning. By explicitly labeling skills, the model's flexibility might be reduced, potentially limiting its ability to generalize to unseen tasks or novel problem types that do not fit neatly into predefined skill categories.\n\nThe related work section is very short and the references are bulk listed.\n\nSome references are incomplete (e.g., [16], [20], [32], [33]). Please fix. Most of these are missing information about where the paper was published. If ArXiv note it, if already published, note that instead.\n\nStray bracket ) on line 119. Stray space after GPT-4-0613 on line 135. \n\nSome acronyms are repeatedly introduced:  e.g., Chain-of-Thought (CoT) [four times]; program-aided language models (PALs)   [multiple times]. \n\n[32] talks of complex skills — which are composed of lower-order skills? How does the current work handle such complex hierarchical skill definitions.\n\nSKILL-MIX may be referred as well — https://arxiv.org/pdf/2310.17567. Describe how your work corresponds to a mixture of skills rather than just a single skill categorization (or at least discuss the implications of this in your work)\n\nFigure 1 uses two alternative notations: e.g., Skill 2 and Skill II. Is there a difference? If not, just use one notation consistently.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed review. \n\n**Multiple skills per question**\n\nThe reviewer has asked for preliminary experiments showcasing multiple skill labels. We refer the reviewer to the common rebuttal for this experiment. We show decent improvements on the MATH dataset with a multi-skill approach. \n\n**Cases where categorization of skills is difficult**\n\nThe reviewer has questioned how the proposed approach would work for cases where skill categorization is difficult. We address this by showing that the proposed approach can be utilized for skill labeling in many different domains beyond MATH. In the common rebuttal, we have used the proposed approach can be used to assign skill labels to a variety of domains spanned by the alpaca, just-eval, hh-rlhf datasets such as journalism, art, literature, coding, engineering, etc. Therefore, we can conclude that as long as a given domain is represented in the LLM training data, we can obtain decent skills for that domain.\n\n**On curation of the labelled dataset being resource intensive**\n\nThe reviewer has mentioned that it may be very resource intensive to create the skill exemplar repository. We find that the computational complexity of creating and storing the repository is O(N) each where N is the size of the dataset since it requires one pass over the dataset. This is not very intensive as it is linear in the dataset size.\n\n**On the flexibility or generalizability of the labeled skills**\n\nThe reviewer has pointed that given the skill exemplar repository, it may be difficult to generalize to tasks that require different capabilities from those represented in the repository. We address this by pointing to experiments which highlight the generalizability and flexibility of the skill exemplar repository. \n\n1. In Table 6, we have transferred the skills derived from the GSM8K dataset to 6 other datasets which encompass various problem types and formats that are not present in the original GSM8K dataset. For example, SVAMP [1] presents problems with certain linguistic variations which are specifically designed to confuse the LLMs. Such kind of confusing problems are not present in the GSM8K datasets. \n2. In Table 4, we have shown that even though the skill exemplar repository does not contain any code generation examples, sourcing skill-based exemplars from it improves code generation performance of LLMs.\n\n**On complex hierarchical skill definitions**\n\nThe reviewer has asked how the proposed approach can handle hiearchical skill definitions such as the compositions mentioned in [1]. \n\nHandling of skill hierarchies is a  complicated problem therefore it is beyond the scope of the current paper. At the moment, our framework supports labeling questions with single and multiple skills as we have shown in this rebuttal. As future work, we hope to extend support to skill hierarchies. \n\n[1] A theory of Emergence of Complex Skills in Language Models\n\n**On Skill-Mix paper**\n\nWe have cited skill-mix as citation number [31] in our work. Skill-mix introduces an evaluation benchmark which evaluates the ability of a LLM to generate text consisting of K skills. On the other hand, this work focuses on extracting 1 or more mathematical skills from text and using them to improve mathematical reasoning in LLMs. \n\n**Figure 1 uses two alternative notations**\n\nSkill 2 refers to the pre-clustering skill and Skill II refers to the post-clustering skill. The diagram indicates that the 5 pre-clustering skills were clustered to 3 aggregate skills. \n\n**How have you determined how many coarse-grained categories to have for the skills — and how do more or less skill categories affect performance?**\n\nThere are no hyperparameters used in the skill labelling, skill clustering, or the skill relabelling phases. We only use the prompts given in App. Figure 2 and let the LLM decide the granularity of skills and the number of clusters. \n\n**Why are results not shown separately for CoT in Table 1? Could CoT not be applied separately to these kinds of questions.**\n\nCoT can be applied separately to the question in MATH. However, we took the CoT result from [1] which does not report the per-topic performance of CoT. \n\n[1] Skills-in-Context Prompting: Unlocking Compositionality in Large Language Models \nhttps://arxiv.org/abs/2308.00304\n\n**How would the results be (let’s say in Table 7), if we identify multiple skills as being relevant to the question at hand [instead of just one].**\n\nWe have presented the modified solutions when using multiple skill labels per example in the pdf provided in the common rebuttal. Both questions are now solved correctly. \n\n**it is a bit of a stretch to call the ability to categorize as a metacognitive ability of the LLM**\n\nWe appreciate the reviewer's succinct definition of metacognition, which corresponds with our understanding. The comment sparked a discussion among authors, leading us to mostly agree with the reviewer's \"stretch\" comment. While metacognitive knowledge can be in many forms, in this work we are interested in the specific knowledge about the catalog of skills required to solve math problems. We think of skills as _pointers_ to knowledge, not the knowledge itself. We could switch to using the more generic term 'metaknowledge'. We welcome the reviewer's further thoughts on naming.\n\nWe find that LLMs can categorize mathematical data into skills and use them to improve their own reasoning. That being said, the proposed approach cannot extract the various other forms of metacognitive knowledge as of now and we hope to explore that in future work. \n\n**Metacognitive knowledge vs theory of mind**\n\nMetacognition involves awareness and control over one’s own cognitive processes, while theory of mind involves understanding others' thoughts and beliefs. Our work focuses on metacognitive knowledge, specifically the ability for an LLM to recognize, label, and apply appropriate skills to tasks to improve its **own** reasoning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewer for their feedback and comments. In this common response we present new experimental results which we obtained during the rebuttal phase which help clarify some of the questions and concerns raised by the reviewers:

## Multi-Skill Labelling
Reviewers ADDk and xadG have mentioned that labelling a question with a single skill is a limitation. We agree with the reviewers that it is a limitation as we have also identified in Section 5. We conducted an initial investigation exploring multi-skill labelling and have demonstrated its potential to improve performance.

We consider the MATH dataset for this experiment. We follow the following steps to adopt a multi-skill setup - 

1. Creating the Skill Exemplar Repository
    - Skill Labeling - We adopt the prompt in Figure 2 (left) and modify it to instruct the model to output multiple skills required to solve the question. 
    - Skill Clustering - We follow a slightly different approach for clustering. We pass the list of skills to the LLM and ask it to combine common skills into a single representative skill. We keep doing this iteratively until there are total N skills remaining in the repository. N is an hyperparameter which is set to be 150. We do not experiment much with this hyperparameter due to lack of time during the rebuttal phase. 
    - Skill Relabelling - Next, we relabel all questions with skills from the clustered list. Each question is labelled with multiple skills. 
2. Inference
    - During inference we label the the inference questions with multiple skills from the clustered list of skills from the repository and fetch K in-context examples with most overlap from the Skill Exemplar Respository. 

We present examples of questions and labeled skills in Table 1 in the attached PDF. 

Here are the results on the Math dataset. We adopted the same setup as Table 2 from the paper which uses GPT-4-0613 and 4 in-context examples:

| **Prompting**                              | **Pre-Algebra** | **Geometry** | **Inter-Algebra** | **Algebra** | **Probability** | **Pre-Calculus** | **Num. Theory** | **Overall** |
|--------------------------------------------|-----------------|--------------|-------------------|-------------|-----------------|------------------|-----------------|-------------|
| **CoT + Skill-Based**                       | 74.28           | 41.75        | 27.02             | **73.12**   | **58.01**       | 33.70            | **51.10**       | 53.88       |
| **CoT + Skill-Based (multiple skills)**     | **79.90**       | **45.93**    | **30.12**         | 71.01       | 53.38           | **38.09**        | 49.07           | **55.14**    |

We can see that the multiple skill approach achieves best overall performance. 

Coincidentally, both the incorrect cases from App. Table 13 are solved correctly now with this approach as shown in Table 2 of the attached PDF.

There are definitely cases where the model fails with errors such as incorrect skill application or calculation mistakes. We omit this discussion here due to lack of space. We will update the paper with this result and corresponding analysis. 

## Empirical evidence that the proposed method is domain agnostic

Reviewer xadG pointed out that we have not demonstrated that the method is domain agnostic despite our claims otherwise. We address this concern by applying the proposed method to two domains: Alignment and SQL queries. 

### Alignment

We consider the problem of alignment via in-context learning [1]. Fetching relevant in-context examples can help LLM provide more helpful answers. 

To apply the proposed approach in this setup we curate a skill exemplar respository of 5000 examples from the alpaca dataset [2], 1000 examples from the lima dataset [3], and 5000 examples from hh-rlhf red team dataset [4] using the same prompts mentioned in Figure 2. Next, we label the examples in the just-eval dataset with skills from the skill-exemplar repository using the prompt shown in Figure 2 (right). We present examples from the skill exemplar repostory in the attached PDF table 3. 

Next, for answering each question in the just-eval dataset, we retrieve 3 in-context examples of the same skill as the question. For the baseline, we sample random examples from the skill-exemplar repository. The results on **Mistral-7B** are as follows. The metrics are calculated using GPT-4 following the approach used in [1]:

|  | helpfulness | clarity | factuality | depth | engagement | safety |
| -- | ------ | ----- | ---- | ----- | ---- | --- |
| CoT + Random | 3.61 | 4.33 | 3.77 | 2.55 | 2.90 | 3.65 | 
| CoT + Skill-Based | **3.73** | **4.40** | **3.89** | **2.64** | **3.01** | **3.78** |

We can see from the table that providing relevant in-context examples based on skills also improves performance in the context of alignment. 


[1] The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning - https://arxiv.org/abs/2312.01552

[2] alpaca dataset - https://huggingface.co/datasets/tatsu-lab/alpaca

[3] Lima dataset - https://huggingface.co/datasets/GAIR/lima

[4] HH-RLHF dataset https://huggingface.co/datasets/Anthropic/hh-rlhf
 
### Generating SQL queries

Next, we consider the task of generating SQL queries. Given a command, the LLM must output a SQL query which satisfies the command. We compare the proposed approach to the Skill-KNN model [1]. Following [1], we consider the **spider dataset** [2] for this task. The skill discovery pipeline is same as for the other experiments in the paper. Here also we use 4 in-context examples similar to [1]. We use the **GPT-4-0613 backbone** for this experiment. Here are the results:

| Method | Acc | 
| --- | --- |
| Skill-KNN (their best performing model) [1] | 82.7 |
| Skill-Based (this paper) | **84.3** | 


[1] An, Shengnan, et al. "Skill-Based Few-Shot Selection for In-Context Learning." 
arXiv preprint arXiv:2305.14210 (2023).

[2] Spider Dataset -=https://huggingface.co/datasets/spider

Meta-review: This submission introduces a novel method for enhancing mathematical reasoning in LLMs through the extraction and use of metacognitive knowledge. The reviewers and I are in agreement about the novelty of the approach and the strength of the empirical results demonstrated across different datasets. We are particularly happy with the paper’s potential to improve the reasoning capabilities of LLMs by using skill-based exemplars during the inference phase.

The primary concerns about this paper are the initial limitation of assigning only one skill per math question and the lack of demonstration that the method is domain-agnostic. The authors have addressed these concerns in their rebuttal by extending their methodology to include multiple skill labels per question and by successfully applying the method to domains beyond mathematics, such as alignment and SQL query generation. This addresses the scalability and generalization concerns to a considerable extent. The concerns about computational complexity raised by the reviewers have also been clarified by the authors, explaining that their method operates with linear complexity during the skill extraction and clustering phases and constant time during inference.

Overall, this is an interesting and novel paper that makes a strong contribution to the field, and as such, I recommend its acceptance.
Decision: Accept (poster)