AdaptMI: Adaptive Skill-based In-context Math Instructions for Small Language Models
Authors: Yinghui He
Abhishek Panigrahi
Yong Lin
Sanjeev Arora
Venue: COLM 2025 (2025)
Keywords: Small language models
large language models
in-context learning
natural language processing
TL;DR: We introduce AdaptMI, an adaptive approach to selecting skill-based in-context math instructions for small language models.

Abstract: In-context learning (ICL) allows a language model to improve its problem-solving capability when provided with suitable information in context. Since the choice of in-context information can be determined based on the problem itself, in-context learning is analogous to human learning from teachers in a classroom. Recent works (Didolkar et al., 2024a; 2024b) show that ICL performance can be improved by leveraging a frontier large language model’s (LLM) ability to predict required skills to solve a problem, popularly referred to as an LLM’s metacognition, and using the recommended skills to construct necessary in-context examples. While this skill-based strategy boosts ICL performance in larger models, its gains on small language models (SLMs) have been minimal, highlighting a performance gap in ICL capabilities. 
We investigate this gap and show that skill-based prompting can hurt SLM performance on easy questions by introducing unnecessary information, akin to cognitive overload. To address this, we introduce AdaptMI, an adaptive approach to selecting skill-based in-context Math Instructions for SLMs. Inspired by cognitive load theory from human pedagogy, our method only introduces skill-based examples when the model performs poorly. We further propose AdaptMI+, which adds examples targeted to the specific skills missing from the model’s responses. On 5-shot evaluations across popular math benchmarks and five SLMs (1B–7B; Qwen, Llama), AdaptMI+ improves accuracy by up to 6% over naive skill-based strategies.

Published as a conference paper at COLM 2025
AdaptMI: Adaptive Skill-based In-context Math Instructions for Small Language Models
Yinghui He Abhishek Panigrahi Yong Lin Sanjeev Arora Princeton Language and Intelligence, Princeton University in0e68@princeton.ede
Abstract
In-context learning (ICL) allows a language model to improve its problem- solving capability when provided with suitable information in context. Since the choice of in-context information can be determined based on the problem itself, in-context learning is analogous to human learning from teachers in classroom. Recent works (Didolkar et al., 2024a;b) show that ICL performance can be improved by leveraging a frontier large language model's (LLM) ability to predict required skills to solve a problem, popularly referred as LLM's melacognition, and using the recommended skills to construct necessary in-context examples. While this skill-based strategy boosts ICL performance in larger models, its gains on small language models (SLMs) have been minimal, highlighting a performance gap in ICL capabilities.
We investigate this gap and show that skill-based prompting can hurt SLM performance on easy questions by introducing unnecessary information, akin to cognitive overload. To address this, we introduce AdaptMI, an Adaptive approach selecting skill-based in-context Math Instructions for SLMSs. Inspired by cognitive load theory from human pedagogy, our method only introduces skill-based examples when the model performs poorly. We further propose AdaptMI+, which adds examples targeted to the specific skills missing from the model's responses. On 5-shot evaluations across popular math benchmarks and five SLMs (1B-7B; Qwen, Llama), AdaptMI+ improves accuracy by up to 6% over naive skill-based strategies.
1 Introduction
Human learning is primarily feedback driven (Hattie & Timperley, 2007; Bandura & Walters, 1977). The most common example is how students refine their understanding on a subject through adaptive examples and feedback from teacher in classroom setting. In the domain of language models(Vaswe) Grattafiori et al., 2024), in-context learning (ICL) (Brown et al., 2020) plays an analogous role. ICL enables models to adapt their problem-solving strategies by conditioning on additional task-relevant information provided in context, possibly sourced from more capable frontier model acting as teacher.
et al., 2017; Achiam et al., 2023; Team et al., 2023;
However, ICL is known to be emergent property (Wei et al., 2022), with larger models showing better ICL capabilities than smaller ones. Small Language Models (SLMs) often struggle to generalize from in-context examples and highly sensitive to how the context is constructed, which limits their ability to learn effectively from in-context instructions. This paper aims to improve the ICL performance of SLMs through careful selection of in-context math instructions.
We build on skill-based in-context example selection from Didolkar al. (2024a;b). This work leverages the metacognitive abilities of frontier large language models (LLMs) to
Code available at: https:/ htops://giii wwi.thu..ccmm hatuu/.cooprrinneetopppp//aaaaa

---

Published as a conference paper at COLM 2025
Stage 1
Stage 2
Adaptive Math Instructions
SLM
Which skills did the wodel fuil feverage in in solation?"
wissing skills
(1) factorization
response
( number theory"
AdaptMI+
LLM
Difficult questions
Reward Model
Skill-based few-shot
Difficult question
AdaptMI
+
low reward
Failure
Skill
Easy questions
Skill
high reward
+ Fixed few-shot
Easy question
Skill 3
Success
Skill Bank
AdaptMI and AdaptMI+ are 2-stage adaptive in-context example selection meth- ods. In the first stage, questions are classified as easy and difficult using a reward model on the SLM's responses and threshold-based filtering. In the second stage, AdaptMI uses skill-based in-context examples only for difficult questions. For AdaptMI+, we use an LLM to identify the key skills missing in the SLM's responses for difficult questions and use specific in-context examples targeted towards the missing skills. For easy questions, we use fixed set of in-context examples. We follow Didolkar et al. (2024a) to get the Skill Bank, skill annotations for each question, and relevant in-context examples for each skill.
Figure 1:
predict the high-level skills required to solve a given task. For instance, when presented with question "3+5*2=", a frontier LLM might infer that the relevant skills are "addition" and "multiplication." " After annotating pool of examples with these skill labels, in-context examples are selected at inference time by first predicting the required skills and then retrieving matching examples. This approach aligns with cognitive theories of human learning that encourage teaching through appropriate skill-based guidance (Kirschner et al., 2006; Sweller, 2011). While skill-based in-context selection significantly boosts the ICL performance of larger models, it fails to improve ICL performance in SLMs.
Ablation reveals an important insight: Skill-based strategy can hurt performance of an SLM on easy questions-ther
that an SLM can already solve without skill-based guidance. We define easy questions for an SLM using a reward model, illustrated in Figure and formally defined in Section 2.2. Across 5 SLMs on the MATH dataset (Hendrycks et al., 2021), we observe an average 4% performance drop on easy questions when using skill-based selection, compared against non skill-based in-context selection strategies. Further fine-grained analysis in Section 4.1 reveals that this strategy leads to long and erroneous responses on easy questions, mirroring overthinking patterns observed in weaker human students when overloaded with information (Diaconis & Mazur, 2003; Liu et al., 2024b).
Core Contribution: Motivated by Adaptive Teaching (Randi, 2022) and Cognitive Load The- ory (Sweller, 2011) which suggest humans learn more effectively when guided specifically tasks they find challenging, we propose AdaptMI. It is an adaptive 2-stage in-context selection method that applies skill-based example selection only to difficult questions where the small language model (SLM) struggles. Additionally, mirroring how humans benefit most from feedback on their mistakes, we further propose AdaptMI+, where the specific skills that are missing from the SL.''s responses are used to create in-context examples. We provide method overview in Figure 1, and outline the details in Sections 2.2 and 2.3.
Our experiments (Section 3) on popular math datasets show that AdaptMI+ can significantly improve the performance of all five tested SLMs by up to 6%, while AdaptMI also yields notable gains of up to 3.6%. On top of that, in Section 3.3, we extend AdaptMI+ to an iterative loop of adaptive example selection and demonstrate its potential of progressively, constantly guiding small language models to tackle harder problems. In Section 4, we provide thorough discussion on why our adaptive example selection method is superior
2

---

Published as a conference paper at COLM 2025
to naive, non-adaptive skill-based selection. We further present several ablation studies on choices of in-context examples and reward model settings. Finally, we discuss future directions in Section 6.
2 Designing AdaptMI and AdaptMI+
2.1 Preliminary
We study k-shot in-context learning with small language models. Let Q be the set of evaluation questions, and let q Q denote question drawn from this set. We consider the k-shot setting, where, given a pool P of question-answer pairs, k examples are selected and included in the prompt for each evaluation question 9. There are two common strategies to select the in-context example pairs.
Fixed k-shot examples: We fix set of k examples from P and use them for inference all evaluation questions, Our experiments will use the examples used by Qwen models for evaluation (Yang et al., 2024).
Random k-shot examples: We utilize k randomly selected examples from the pool of in-context examples P for each evaluation question.
Our work builds on skill-based in-context selection from Didolkar et al. (2024a), which we describe here. While hard to define precisely, a skill is informally defined as basic computation necessary to solve a task at hand. For example, necessary skills to solve arithmetic tasks could be addition, subtraction, multiplication and division. We will use Skill-bank( Q), set of skills that are necessary to solve questions in Q. These skills are enlisted from a large model like GPT-4 using an appropriate prompting strategy et al., 2024a; Kaur et al., 2024). Next, each question in the evaluation set Q and the in-context example pool P are matched to the necessary skills from the Skill-bank(44 (Q). We will use Skill-Map QUP + Skill-bank( Q)* i to refer to the map between each question to set of k skills, which we will get by prompting an LLM (Achiam et al., 2023). Then, skill-based in-context examples are decided as follows:
Stradlgy(Didolkar (Didolkar
Skill-based k-shot examples: For each question 9 E 2, we pick a set of k examples using Ski)I-Map(q), by randomly picking one example for each skill in Skill-Map(g). More formally, for each skill s in Skill-Map(q), we randomly pick an example from the pool of in-context examples P which is annotated with the skill and return the union of the selected examples for all the skills. This is formally outlined in Algorithm in appendix.
Now, we define AdaptMI and A built on the above-defined prompting strategies, that consist of primary stages. Section 2.2 formally introduces the first stage that identifies easy and difficulf questions for an SLM using reward model. Section 2.3 then presents the prompting strategy for AdaptMI and AdaptMI+ on easy and difficult questions.
2.2 Stage 1: Detection of easy and difficult questions via reward filtering
In this stage, we will label question 9 E as easy or difficult for an SLM. We could simply define difficult questions those set of questions that the model gets wrong with fixed or random k-shot prompting. However, this requires access the ground truth labels. Instead, to make our technique more broadly applicable, we use reward model to classify the responses of the SLM. The reward model need not be a perfect reward model, we give more details in Appendix B.1. Given question 9, use reward model on the response of the SLM when prompted with fixed k-shot examples.
Scoring with process reward model: Because we primarily focus math datasets, we assume that the model's response is composed of steps for a question q and contains answer in its final step. We will use the reward model to output reward scores for each step. For simplicity, we will refer to the scores of the reward model as {rg.1.-
.. ..ghh Then, we
use thresholds T T2 to filter out difficult questions for the SLM. We will refer to the threshold filtering function as R
{0,1) 1).
3

---

Published as a conference paper at COLM 2025
0, (if) g I &
(final step has low reward)
1 (or)
} - y S 71
(average low reward across all steps)
((q) =
(1)
(or) 3i < f s.t. qi S 12
(low reward at any step)
1, otherwise,
Difficult vs. easy questions. We define Qdifficult as the set of questions with low-reward model responses R. Accordingly, Ceasy denotes all remaining questions.
Qdifficult = [9 (q) = 0}
Qeasy = {q R(q) = }}
(2)
2.3 Stage 2: Skill-based selection of in-context examples
AdaptMI uses skill-based k-shot examples for difficult questions and fixed k-shot examples for easy questions.
AdaptMI: For difficult questions Qdifficult use skill-based k-shot examples. For easy question Qeasy, we use fixed k-shot examples.
Instead of using in-context examples for all skills relevant to a difficult question, AdaptMI+ focuses only the skills that the model's initial response lacks:
AdaptMI+: For each difficult question 9 in Qaifficult. we use large LLM (GPT-4o-mini) to predict the set skills in Skill-Map(q) that are missing in the model's response. Then, for each skill that are missing, we randomly pick an example from the pool of in-context examples P which is annotated with the skill s and return the union of the selected examples for all the missing skills. For easy questions Qeasy, we use fixed k-shot examples.
Experiment
3.1 Experimental Settings
Datasets. We evaluate on the MATH (7.5k training samples and 5k test samples) (Hendrycks et al,, 2021) and GSMSK (7.4k training samples and 1.3k test samples) (Cobbe et al., 2021) datasets. We follow Didolkar et al. (2024a) to label skills on both the training and test sets using GPT-4o-mini (OpenAI, 2024), and run inference experiments on the whole test set. Appendix A.1 shows the prompt and examples of our skill annotation pipeline. We sample in-context examples from the training set. These two datasets are not overly challenging for SLMs, which ensures relatively interpretable model outputs for stable failure detection. Meanwhile, they are sufficiently representative to offer meaningful insights into our method's is efficacy.
Model settings. We tested our methods on five instruction-tuned small language models: Owenn2.5-..B-Innsruuct, Owen2.5-3B-Instruct, Qwen2.5-7B-Instruu. and Llama-3.23B--Instruu (Yang et al., 2024; Meta AI, 2024). We evaluate the models on 5-shot ICL performance. We use generation temperature also compare against consistency@5 voting (Wang et al., 2022) with 5-shot fixed examples, where we use 5 generations at temperature 1.0 and evaluate the consistent response. For classifying easy and difficult questions in the first stage, we use RLHFlow /Llama3.1-8P PaM-Mistral-Data (Xiong et al. (2024)), an 8B process reward model fine-tuned from Llama-3.1 with filtering thresholds T 0.85, T2 0.7. We use GPT-4o-mini for skill annotation as well as labeling missing skills in AdaptMI+.
dama-3.2-1B-Inntruue
0.0 for all experiments. We
method to non-adaptive in-context example selection meth- ods, respectively feeding in fixed examples, random examples, and skill-based examples (Didolkar et al. (2024a)) for all queries.
Baselines. We compare
4

---

Published as a conference paper at COLM 2025
MATH Algebra
GSM8K
Methods
Geometry
Precalculus
Prealgebra
Avg.
Avg.
# Biven..--1.8--nnsmmae
Fixed Examples
39.7
38.3
72.2
67.3
52.8
71.5
Random Examples
42.8
41.0
73.1
68.1
53.3
70.9
Skill-based Examples
43.2
39.6
72.0
67.7
53.0
66.1
Consistency@5
44.5
43.5
77.6
70.8
56.9
75.6
AdaptMI
44.7
42.1
76.8
72.0
56.4
72.9
A
44.5
42.1
78.2
72.8
57.2
75.8
Quven2.5 10emm2.5-3BBnsstruc
Fixed Examples
56.4
53.5
85.4
79.7
66.6
84.7
Random Examples
54.7
53.7
85.3
78.9
66.1
84.9
Skill-based Examples
53.4
55.7
86.2
80.7
66.9
85.4
Consistency@5
61.9
55.3
87.4
81.4
68.9
87.0
AdaptMI
54.9
56.2
87.7
81.8
67.8
87.4
AdaptMI+
56.0
55.5
88.3
82.1
69.1
87.7
Queen2.5-7B-nnst
Fixed Examples
61.2
61.5
91.2
87.1
74.7
91.7
Random Examples
60.1
62.1
91.4
86.6
74.4
91.1
Skill-based Examples
61.2
64.3
90.6
87.7
74.4
91.7
Consistency@5
62.4
57.7
92.3
87.0
75.1
93.3
AdaptMI
62.2
64.7
91.5
87.6
75.9
92.3
AdaptMI+
64.9
63.4
92.8
88.8
76.7
92.4
# -lamm-3.2-11 2-1--.33221B-nnstrcctt
Fixed Examples
8.0
11.1
19.6
21.3
13.8
26.8
Random Examples
10.2
6.5
24.0
20.9
13.7
19.3
Skill-based Examples
14.8
6.8
16.7
22.6
13.4
13.4
Consistency@5
13.6
13.3
28.8
28.2
19.4
29.9
AdaptMI
13.6
10.3
20.8
29.3
16.2
23.2
AdaptMT+
17.1
11.1
29.6
35.4
19.8
26.0
Samma-3.23B--nnttree
Fixed Examples
26.1
29.8
63.8
67.6
46.2
75.8
Random Examples
26.9
61.9
55.3
41.3
76.2
Skill-based Examples
29.6
31.7
66.2
63.3
45.9
71.7
Consistency@5
36.1
23.9
60.0
61.9
44.1
80.7
AdaptMI
28.4
31.7
71.6
71.3
49.8
76.4
AdaptMI+
29.6
35.6
68.1
71.3
49.4
80.7
Table 1: AdaptMI and AdaptMI+ demonstrate a consistent accuracy gain by up to 3.6% and 6% respectively, compared with baseline methods. We present all results as Pass@1 accuracy unless otherwise indicated. Due to space limits, we provide the results on Number Theory, Intermediate Algebra, and Counting Probability Appendix D.
3.2 Performances of AdaptMI and AdaptMI+
Table reports the main results of our adaptive in-context learning method. The baseline methods with non-adaptive in-context examples (fixed, random, or skill-based) results in largely similar Pass@1 accuracy, while consistency@5 can improve accuracy by few percentages. Across all model sizes, our methods AdaptMI and AdaptMI+ consistently outperform the non-adaptive Pass@1 baselines, and par with Consistency@5 perfor- mance on most subareas. The overall improvements are especially pronounced for smaller models, @wwen..5-1.5B. 1.B-IInstruct and 1a-ma-3.2-1B-Insttie.

---

Published as a conference paper at COLM 2025
(Ours) Iterative skill-based selection
(Baseline) Iterative random retrieval
Qwen2.5-1.558 5-5.15551.5-
Qwen2.55 5-3B-Instruct
Qwen2.5- 5-7B-Insttrcct
75.4
68.5
Acouracy (%)
75.2
58
68.0
56
67.5
74.8
54-
67.0
74.6
52
1b
66.5
74.4-
2
10
#lteration
#lteration
#lteration
Figure 2: SLM performances under iterative skill-based example selection (AdaptMI+) vs. iterative random example retrieval. Each iteration involves model inference, difficult ques- tion detection, and random/ /sskll-lssssee example re-selection with GPT-4o-mini Iterative AdaptMI+ yields a continuous accuracy gain by up to 7.2%, while the baseline leads to fluctuated performances.
While AdaptMI surpasses consistency@5 performance most domains, it slightly lags behind on certain subjects such as Geometry and Precalculus for 1B or 3B models. These subjects are relatively difficult for the model, suggested by their loss scores compared to other subjects (see Table 12 in Appendix). Since AdaptMI requires models to have sufficient capabilities leverage the given skill-based examples, may not work better than Consistency@5 on these harder topics.
Notably, A brings significant performance gain across all areas by up to 6,, reflect- ing its strength in accurately targeting model failures. AdaptMI also substantially improves performance by up to 3.6% for 3B-Instruct on MATH. This indicates that our adaptive instruction methods are effective on pererrrerfoninngggg models even without the aid of an L.M.
Llama-3.2-1B-Inns-
and Llama-3.2-
Swwen.5--BBInnstrus and Qwen2.5-7B-Instmee however, AdaptMI shows smaller effectiveness compared to AdaptMI+, This may suggest that higher- performing models require more intelligent and target skill identification process. Overall, these results demonstrate the effectiveness of adaptive example selection and highlight the potential of our approach to elicit the full reasoning capabilities of small language models.
On stronger models such
3.3 Iterative AdaptMI+
Our method can be extended to an iterative loop of adaptive example selection. Each iteration begins with model inference, followed by detecting difficult questions and using GPT-4o-mini to select skill-based examples. The selected examples are then fed in with difficult questions for model inference in the next iteration. This iterative AdaptMI+ is essentially pushing the SLM to tackle a gradually refined set of difficult questions by adaptive teaching. We compare iterative AdaptMI+ with a baseline of iterative random retrieval, where the loop involves inference, random example resampling, and re-inference.
Figure shows that iterative AdaptMI+ consistently improves the reasoning performance on MATH for all three Qwen small language models, while the baseline method struggles to keep pushing the accuracy boundary after the first few iterations. For 1.5B and 3B models, the performance grows rapidly in the first four iterations, and improves more gradually thereafter. The 7B model performance, while starting to degrade by the 10th loop, still increases substantially compared to baseline. Through iterative re-selection of targeted in- context examples, iterative AdaptMI+ demonstrates the potential of progressively guiding small language models to tackle unsolved problems.

---

Published as a conference paper at COLM 2025
Question Difficulty
Example Type
MATH Algebra
GSM8K
Geometry
Precalculus
Prealgebra
Avg.
Avg.
Fixed Random
21.3
23.7
44.8
35.1
29.8
45.2
23.2
25.3
53.9
40.5
31.2
46.1
Difficult
28.4
28,9
55,1
45,5
35.7
48.0
Skill-based
+7.1
+5.2
+10.3
+10.4
+5.9
+2.8
Fixed
82.1
81.8
94.6
93.7
90.2
96.3
Random
81.6
78.9
92.1
92.3
87.6
90.6
Easy
77.2
71.5
85.9
86.0
81.0
83.2
Skill-based -4.9
-10.3
-8.7
-7.7
-9.2
-13.1
Table 2: Accuracy of Dwee2.5511.B-Innttie
difficult and easy questions, respectively under fixed, random, and skill-based examples. Skill-based examples boost performance difficull questions across all categories, while significantly underperforming questions. We provide the results on Number Theory, Intermediate Algebra, and Counting & Probability, as well as the results on other Qwen models in Appendix D.
easy
4 Discussion
4.1 Why does adaptive selection work better than non-adaptive skill-based selection?
To better understand, we compare performance under fixed, random, and skill-based in-context examples easy and difficult questions. From Table we observe clear trend that skill-based examples harm an SLM's performance on the set of easy questions, while effectively boosting performance skill-based in-context examples might harm performance on easy questions, we present two illustrative cases where the model's performance regresses when using such prompts.
the difficult ones. To gain deeper insight into how
Case Study 1: Skill-based examples lead the model to overlook key problem constraints. In this example (see Appendix C.1), the Qwen2.5-7B-Instruus model is given an algebra question that includes multiple geometric constraints. When prompted with fixed examples, the model correctly identifies two possible answers and chooses the correct one according to the given condition "both coordinates are negative." On the other hand, when conditioned on examples that represent algebraic skills, the model overly emphasizes algebraic completeness but overlooks this important problem condition. It finally selects the incorrect answer by a random guess.
Case Study 2: Symbol-heavy skill-based examples cause the model to overthink. This question (see Appendix C.2) requires a plug-in-and- approach instead of solving an equation. With fixed in-context examples, the model is able to find out the correct answer by directly plugging in and trying out small values. However, the skill-based examples that involve equation solving may have caused the model to overthink. After failing in the first plug-in-and-ted it ended up attempting to solve the equation system and eventually failed.
4.1.1 Fine-grained Analysis: Effect of skill-based examples across five difficulty levels
The above observations motivate more fine-grained analysis. We partition our evaluation set into five levels of difficulty, based the probability of success under Best-of-n sampling (Gui et al., 2024), verified using ground-truth labels. Formally, a question belongs to Difficulty Level e (1 < < 4) if it can be solved with Best-of-2 sampling, but not with any lower n. Questions that belong to Level can't be solved with Best-of-8 sampling. We provide no in-context examples when measuring the success of Best-of-n sampling and use temperature of 1.0. Intuitively, questions in Level 2 are those where the model is more susceptible to minor issues like formatting, where fixed in-context examples could help. For questions in higher levels, on the other hand, the model might benefit more from guidance with carefully selected in-context examples.
7

---

Published as a conference paper at COLM 2025
Fixed Examples
Skill-based Examples
54.3
350
Difficulty Level
76.1
465
70.4
535
4-
25.0
50.0
452
540
20
40
60
300
400
500
600
700
Accuracy (%6
Average Output Length (Token)
Figure 3: Accuracy and average output length of Qwen2.5-3B -nstruuuutt questions of Difficulty Level 1-5, designed using its Best-of-n performance, with fixed and skill-based examples. Skill-based examples hinder performance on Levels and 2, while helping Levels 3-5. On all difficulty levels, skill-based examples result in noticeably longer outputs.
After splitting the questions into levels, we compare the effect of skill-based in-context examples with fixed in-context examples on the model's responses to questions in each difficulty level. Figure reports the results on a Qwen-3B model and MATH dataset.
Primary observations: We clearly observe that skill -baeed in-context examples can perform worse than fixed in-context examples levels and 2. On the other hand, skill-based in- context examples can substantially help the model on questions in levels 3-5. Furthermore, we observe that responses of the model are substantially longer with skill-based in-context examples, when compared with model responses with fixed in-context examples.
This shows that with skill-based examples, the model can return unnecessarily longer responses and make mistakes easier questions, when simple strategies like Best-of-2 sampling or prompting with fixed in-context examples would have sufficed. This aligns with existing works the issues of longer mhain-of-thoughe reasoning in language models and how it relates to "problems of over-thinking" in humans (Liu et al., 2024b; Diaconis & Mazur, 2003).
4.2 Ablation Studies
Effect of in-context example choices in Stage 2. Our main method combines difficult questions with skill-based examples and easy ones with fixed examples, based on the observation that models only need targeted instructions better understand its effectiveness, we conduct an ablation study exploring alternative combinations of in-context examples. Our primary observations
more challenging cases. To
As shown in Figure 4, our combination of dlifficult-skill-baale efse- consistently outperforms all other configurations. Notably, the accuracy gap between the best and worst-performing combination choosing in-context examples for SLMs.
reach 7.1%, which stresses the importance of carefully
The sensitivity to in-context example selection varies across model sizes, with the 1.5B model being the most sensitive and the 7B model being the most stable.
Effect of threshold values the reward model prediction. We investigated the effect of Tj and T2 (defined in Section 2.2) on the classification performance of easy or difficult questions. Specifically, we measure whether our classification of questions as easy or difficult also corresponds to the correctness of responses assessed using ground-truth labels. In
We also present results using the difficulty split of questions annotated in the original MATH dataset in Appendix B.3, Differences in performance and generation length of model's responses with skill-based and fixed in-context examples are less pronounced across difficulty levels. This is expected, model's own responses must be better fine-grained indicator on question difficulty.
8

---

Published as a conference paper at COLM 2025
Qwen2.5-14 10..551..--Inntttue
Qwen2.5-3B-Inae ----sruuttt
Qwen2.5-76 500.-----nnntt
564
Fixed Examples
52.8
51.8
49.3
66.6
66.3
65.7
74.7
74.4
73.2
Difficult
67.3
75.2
Random Examples
53.7
53.3
50.2
66.9
66.1
65.9
66.8
74.8
74.4
73.2
74.6
Skill-based Examples
56.4
53.3
53.0
-51.1
67.8
67.5
66.9
75.9
75.6
74.4
73.9
49.3
65.7
73322
Pixed
Random Easy
Skill
Fixed
Random
Skill
Pixed
Random Easy
Skill
Easy
Figure 4: ICL performance, measured in terms of accuracy, across different combinations of in-context examples for easy and difficult questions the MATH dataset. Across all models, we observe that skill-based in-context examples for difficult questions and fixed in-context examples for the easy questions work the best.
Table we report four metrics (accuracy / precision recall F1) evaluating the prediction accuracy resulting from different filtering thresholds. Note that T 0 or T2 = 0 means completely removing the constraints of Ti or 72. Across all evaluated combinations of threshold values, our choice of the threshold values (Tı = 0.85, T2 0.7) gives a good combination of prediction scores. To further visualize this effect, we conduct AdaptMI on top of all combinations of thresholds, and report the final accuracy in Table 4. Our choice of threshold values yields the highest final accuracy among all the combinations.
Ti 12
0.6
0.7
0.8
53
00/00/0
0/0
80
/ 78
79
79
80
74
88
79
75
66
95
78
T 0.8
80
79
78
79
80
76
85
80
79
72
90
80
75
66
78
T 0.85
79
/
74
/
88
/
80
79
/
72
90
78
70
/
92
/
65
96
T 0.9
73
/
64
95
77
73
/ 64
95
77
72
/
64
96
77
70 / 62
97
75
Table 3: Reward model performance (accuracy
precision recall F1) on classifying
correct/
correct/incorrect incorrect responses from 1rren..51..B-Imstrrue on MATH, accross different thresh-
olds. Ti 72 0 means completely removing T or T2. Our choice of threshold values (7 0.85, T2 = 0.7) gives a good combination of prediction scores.
12
T2 =
0
0.6
T2
0.7
0.8
Ti =
52.8
55.7
55.9
55.7
Ti 0.8
55.1
56.3
56.2
55.6
T 0.85
55.3
56.4
56.4
55.6
T 55.7
55.7
55.6
55.2
Table 4: Final AdaptMl performance of Qwwe2.5-1.5BIInstru thresholds. Our choice of threshold values (T) 0.85, T2 = 0.7) leads to the highest accuracy.
MATH, with different
Additional ablations. We compare a process reward model with an outcome reward model in Appendix B.1. We further show the potential of using alternate heuristic filtering methods to use in place of reward models to classify easy and difficult questions. We find that these heuristic strategies could replace reward models with appropriate hyperparameters. We keep full exploration to future work. We also explore an alternative strategy to construct adaptive in-context instruction, where feed in natural language instructions provided by LLM in place of in-context examples, in Appendix B.2. We find that the models simply ignore in-context information that contain long, and unstructured natural language feedback.

---

Published as a conference paper at COLM 2025
5 Related Works
In-context learning example selection. As key feature of language models, the in-context learning ability (Brown al. (2020)) enables models to improve performance without undergoing gradient-based training. This ability can be maximally activated with carefully chosen in-context demonstrations. Prior works have extensively studied the dynamics of in-context learning (Chen et al. (2024)) and effective techniques of in-context example selection (Zhang et al. (2022); Cheng et al. (2023); An al. (2023); Didolkar et al. (2024a); Liu et al. (2024a)) for larger models (>13B). These heuristics often simply rely on the semantic relation between the question and examples, and they typically require training a dedicated example selection model. Meanwhile, the in-context learning dynamics of small language models are understudied.
Classifying model failures. Identifying and understanding language model failures helps us adaptively improve model performance, e.g., via targeted training data selection (Zeng et al. (2025)). Prior works have utilized models' test-time failure patterns build adaptive datasets with difficult questions (Dinan et al. (2019); Nie et al. (2020); Ribeiro & Lundberg (2022); Gao et al. (2023); Li al. (2025)). However, these failure identification and classifica- tion approaches have rarely been applied to inform in-context example selection.
Symbolic and Skill-based Reasoning. Performing syntoooiigggmbooiiccccoooc rebbooinn can largely enhance language models' math reasoning ability (Sullivan & Elsayed (2024); Alotaibi et al. (2024); Xu et al. (2024); Shaik & Doboli (2025)). As SLMs generally possess weaker capabilities to understand complex in-context information, symbolic knowledge aids SLM reasoning by providing structured, less-noisy contextual information (Liao et al. (2024)). Notably, the concept of "skill" was proven effective as a useful criterion for clustering symbolic knowledge (Didolkar et al. (2024a)), guiding contextual example selection (Didolkar et al. (2024a); An et al. (2023)) and mixture-of offeuress routing (Chen et al. (2025)).
6 Conclusion
Our work explores reasons behind the failure of skill-based in-context examples to boost ICL performance of SLMs. We show that skill-based selection can make the model "overthink" on easier questions, which leads to a degradation in ICL performance. We then propose adaptive in-context selection strategies, AdaptMI and AdaptMI+, that use skill-based selec- tion only for difficult questions.
While our primary focus is on improving ICL, performance in SLMs, an important question is whether similar strategies can also guide the training of better SLMs. Current approaches often rely on distilling (Hinton et al., 2015) an SLM directly from the logits or generations of frontier LLM, which requires careful curation of training data and training pipeline for optimal and efficient benefits (Hsieh et al., 2023; Ivison et al., 2023; Kaur et al., 2024). Recent studies suggest that additional in-context information can help models learn more effectively or efficiently. However, these strategies employ static or manually crafted curricula and in-context information (Zhu et al., 2025; Gao et al., 2025; Liao et al., 2024; Allen-Zhu & Li, 2024), An important open direction, thus, is how to adapt AdaptMI and AdaptMI+ to enable SLMs to train more effectively using frontier LLMs.
Acknowledgements
We thank the members of Princeton Language and Intelligence for their helpful discussion and feedback. Sanjeev Arora and Abhishek I'anigrahi are funded by NSF, Darpa, ONR, and Schmidt Foundation. Abhishek Panigrahi is a current Apple AIML scholar.
10

---

Published as a conference paper at COLM 2025
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. Xrv preprint ar Xrx::23033008774 2023.
Zeyuan Allen-Zhu and Yuanzhi Li.
Physics of language models: Part 3.3, knowledge capacity scaling laws. arXiv preprint ar arXio:2404.05405, 2024.
Fatimah Alotaibi, Adithya Kulkarni, and Dawei Zhou. Graph of logic: Enhancing Ilm reasoning with graphs and symbolic logic. In 2024 IEEE International Conference on Big Data (BigData), pp. 5926-5935. IEEE, 2024.
Shengnan An, Bo Zhou, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng, Weizhu Chen, and Jian-Guang Lou. Skill-based few-shot selection for in-context learning, 2023. URL https://arxiv.orgg/ www.:oog//////... 2305. 14210.
Albert Bandura and Richard H Walters. Social learning theory, volume 1. Prentice hall Englewood Cliffs, NJ, 1977.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Serbert-Vosss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz: Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Tlya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL httpp:://aaat
herrg/bb//2000/ 14165.
Justin Chih- Chen, Sukwon Yun, Elias Stengel-Eskin, Tianlong Chen, and Mohit Bansal. Symbolic mixture-of-experts:: Adaptive skill-based routing for heterogeneous reasoning, 2025. URL htttp:://arxiv.orgg/a/. otg/oos// abeaaba.../...///////// 05641.
Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, and He He. On the relation between sensitivity and accuracy in in-context learning, 2024. URL https://arrxvvorr//ba 07661.
Daixuan Cheng, Shaohan Huang, Junyu Bi, Yuefeng Zhan, Jianfeng Liu, Yujing Wang, Hao Sun, Furu Wei, Denvy Deng, and Qi Zhang. Uprise: Universal prompt retrieval for improving zero-shot evaluation, 2023. URL htttt://////vi
htttts:://arrvv.oog/a//////// wwwp.oo///bbbb/3/../ 08518.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. ar arxiv preprint ar Xiv:22210.1 14161....1..... 2021.
Persi Diaconis and Barry C Mazur. The problem of thinking too much. Bulletin of the American Academy of Arts and Sciences,56(3)226 56(3):26-355 2003.
Aniket Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Timothy Lillicrap, Danilo Jimenez Rezende, Yoshua Bengio, Michael Mozer, and Sanjeev Arora. Metacognitive capabilities of llms: An exploration in mathematical problem solving. Advances in Neural Information Processing Systems, 37:19783-19812, 2024a.
Aniket Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Timothy Lillicrap, Danilo Rezende, Yoshua Bengio, Michael Mozer, and Sanjeev Arora. Metacogni- tive capabilities of Ilms: An exploration in mathematical problem solving, 2024b. URL https: htops://arxii www. /tg/ 2b5/ 12205.
Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. Build it break it fix it for dialogue safety: Robustness from adversarial human attack. Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4537-4546, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19 http:://aclaathooooy.oo/////11/1
URL
11

---

Published as a conference paper at COLM 2025
Irena Gao, Gabriel Ilharco, Scott Lundberg, and Marco Tulio Ribeiro. Adaptive testing of computer vision models, 2023. URL https: http:sss/aaax..oo///////22222200
Tianyu Gao, Alexander Wettig, Luxi He, Yihe Dong, Sadhika Malladi, and Danqi Chen. Meta- data conditioning accelerates language model pre-training. Xrv preprint ar wxiv:2501.00955 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The llama 3 herd of models. Xrviv preprint ar arXio:2407.21788, 2024.
Lin Gui, Cristina Gårbacea, and Victor Veitch. Bonbon alignment for large language models and the sweetness of best-of-n sampling. arXiv preprint ar vxXiv22406.00833 2024.
John Hattie and Helen Timperley. The power of feedback. Review of educational research, 77 (1):81-112, 2007.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arviv preprint ar arXi221003.08874 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. Xivi preprint axXto:1503.02531, 2015.
Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexan- der Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. ar Xiv preprint ar orXt:2305.00230) 2023.
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah Smith, Iz Beltagy, et al. Camels changing climate: Enhancing Im adaptation with tulu 2. arXiv preprint ar arXio:2311.10702, 2023.
Simran Kaur, Simon Park, Anirudh Goyal, and Sanjeev Arora. Instruct-skillmix: powerful pipeline for Ilm instruction tuning. arXiv preprint ar Kir:2408.14774, 2024.
Paul Kirschner, John Sweller, and Richard E Clark. Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem- based, experiential, and inquiry-based teaching. Educational psychologist, 41(2):75-86, 2006.
Xiang Lisa Li, Farzaan Kaiyom, Evan Zheran Liu, Yifan Mai, Percy Liang, and Tatsunori Hashimoto. Autobencher: Towards declarative benchmark construction, 2025. URL wwtp:/o///bb////...2/88 08351.
Huanxuan Liao, Shizhu He, Yupu Hao, Xiang Li, Yuanzhe Zhang, Jun Zhao, and Kang Liu. SKIntern: Internalizing symbolic knowledge for distilling better cot capabilities into small language models, 2024. URL https: httpp::/aaxxivorrg/bbss. 2409. 13183.
Haoyu Liu, Jianfeng Liu, Shaohan Huang, Yuefeng Zhan, Hao Sun, Weiwei Deng, Furu Wei, and Qi Zhang. se2: Sequential example selection for in-context learning, 2024a. URL https:/ /ttxiv..vvvvvvv///vv//ee////aa org//bs//402. 138002..1884. 13874.
Ryan Liu, Jiayi Geng, Addison Wu, Ilia Sucholutsky, Tania Lombrozo, and Thomas Griffiths. Mind your step (by step): Chain-of-thouge can reduce performance tasks where thinking makes humans worse. ar preprint arXio:2410.21333, 2024b.
Meta AI.
Revolutionizing Edge AI and Vision with Open, Customizable Models, 2024.
Llama 3.2:
URL https://ai meta. com/blog/
1lama-
12

---

Published as a conference paper at COLM 2025
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial NLE: A new benchmark for natural language understanding, In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Compudational Linguistics, pp. 4901, Online, July 2020. URL
Association for Computational Linguistics. doi: 10. /1/16////1/2222ac/maiiii acl -main.41144
OpenAI. Gpt-4o mini: advancing cost-efficient intelligence. gpt-4-o-min--ddaaciing cost cowpr/cinnnnttt-fffffiie-----iee
2024.
Judi Randi. Adaptive teaching. In Routledge encyclopedia of education, educational psychology. Routledge, 2022.
Marco Tulio Ribeiro and Scott Lundberg. Adaptive testing and debugging of NLP models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3253-3267, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: URL
10.18653/ / 1/l///22cccloong......
202. acl- 230/.
Hashmath Shaik and Alex Doboli. Using a symbolic knowledge graph to address Ilm limitations in analog circuit topology generation. In 2025 JEEE 15th Annual Computing and Communtication Workshop and Conference (CCWC), pp. 00528-00533 IEEE, 2025.
Rob Sullivan and Nelly Elsayed. Can large language models act as symbolic reasoners? arviv preprint ar Xit::2240 204000...2499 2024.
John Sweller. Chapter two cognitive load theory. volume 55 of Psychology of Learn- 10.1111// ing and Motioation, pp. 37-76. Academic Press, 2011. doi: https:/ / URL bttps://wwar http://wwwwwieceeiiieee
B978-0-12- 1--------------9-------- 1887691-1100002. pii/B97801 23876911090028.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. ar Xrv preprint ar xrXi:22322.11803 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances neural informa- fion processing systems, 30, 2017.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. ar preprint ar Xiv:22003.11177, 2022.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXio:2206.07682 2022.
Wei Xiong, Hanning Zhang, Nan Jiang, and Tong Zhang. An implementation of generative 2024.
prm. https: / ht///githuuue
Jundong Xu, Hao Fei, Liangming Pan, Qian Liu, Mong-Li Lee, and Wynne Hsu. Faithful logical reasoning via symbolic chain-of-thought, 2024. URL https://arxiv. http:/wwwwrg///////////// 2405. 18357.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arx preprint arXi1=2412.15115, 2024.
Zhiyuan Zeng, Yizhong Wang, Hannaneh Hajishirzi, and Pang Wei Koh. Evaltree: Profiling language model weaknesses via hierarchical capability trees, 2025. URL https://arxiv. or org/abs/2503. 200.b///500.088..
Yiming Zhang, Shi Feng, and Chenhao Tan. Active example selection for in-context learning, 2022. URL https: /ttp///axvvvv.ee/eeeeeeeeeeee wwriiiiooog//aeee orggabs//2111 04486.
13

---

Published as a conference paper at COLM 2025
Xingyu Zhu, Abhishek Panigrahi, and Sanjeev Arora. On the power of context-enhanced learning in Ilms. Xiv preprint wrXi=:2503.01821, 2025.
Appendix
Table of Contents
Experimental Details
A.1 Skill Annotation MATH and GSM8K
15
A.2 Missing skill Identification from Model Responses
18
A.3 Skill-based Example Retrieval
20
B Ablation Study
21
B.1 Ablations on the reward filtering method in Stage 1
21
Comparing fhw-shot instructions with natural language instructions
22
B.3
Fine-grained analysis of skill-based and fixed in-context examples origi- nal manual split of MATH dataset
C Case Studies
24
C.1 Skill-based examples lead the model to overlook key problem constraints
25
C.2 Symbol-heavy skill-based examples cause the model to overthink.
27
Additional Results
28
Classification results easy and difficult questions
28
D.2 AdaptMI and AdaptMI+ performances
29
D.3 Effect skill-based examples difficult and easy questions
29
14

---

Published as a conference paper at COLM 2025
A Experimental Details
A.1 Skill Annotation on MATH and GSM8K
As described Section 3, we follow Didolkar et al. (2024a) to label skills on both the training and test sets of MATH and GSM8K using GPT--4-mini (OpenAI, 2024). We enlist all skills that we used to annotate the questions in MATH and GSM8K dataset in Tables and and appendix A.1, which have been taken from Didolkar et (2024a). We ask the LLM to read the question and provide up to five skills required to solve this question, from the given existing skill list. We show an example prompt for annotating MATH Number Theory questions as follows.
Example skill annotation prompt for MATH Number Theory questions
[TASK]
You'll be given a math question. Your task to output: (1) < skill> list here up five skill(s) that are required solve this problem,
seperated by commas </skill>,
(2) <reason>> reason here why these skills are needed < </reason>.
[SKILL LIST]
You should only choose the skills from this list:
"arithmetic.sequenne
coavesconverso""",
"basic arithmetic", division and remainders", "exponentiation", "factorization",
"modular aarihmmmiirii "number manipulation", "number theory",
" "polynomial httilyoomiallppraattoonnss ooperation"., pprime_number theory", "sequence.analyae "solving.equateee equations" undeesstadding of fractions"
[QUESTION]
{question}
[REASON AND SKILL(S))
Table 5 shows some example MATH questions and their corresponding annotated skills. From the skill annotation, we construct Skill Bank (see Figure and Section 2.1) that stores the required skills for each question.
15

---

Published as a conference paper at COLM 2025
Question
Annotated skills
What is the units digit of
exponentiation, modular arithmetic, sequence analysis
+ + + 33 + + +... 32009
In the addition problem each letter represents a distinct digit. What the numerical value E? [Figure]
basic arithmetic, number manipulation, solving equations
In triangle ABC, (an(ZCAB) = 77, and the altitude from A divides BC into segments of length 3 and 17. What the area of triangle ABC?
geometry and space calculation, trigonometric calculations, arithmetic operations
Table 5: Example MATH questions, and the annotated skills generated by GPT-4o-mini.
Subject
List Skills
Per subject split in MATH
Algebra
algebraic.eppression..kkll,
arithmetic skills, calculation and conversion skills,
combinatorial operations and basic ari ariethttic,
complex..umber.skilll,, http::iii....dstttc...mmmmmmnnnd
number_theorg thmnerr pol.s/saaoooommmlllaliaaa Lhterryysskklllsspyyooooaaaall,
quadratic equation skills, ratio and proportion skills,
Counting and Probability
combinatooiilllmathemtiic.. cominniiricsssssssccccocc..c.sss knowledge, knowleeede
hrinoingg.riiicipplssaaaatoooooo and prime fat.eoiaattiiiiiion,,
permutattionnandl.ommbiinttooo,
probbillt..ccccaaatoovvttt.....p
coobablltty.cocepppsssa.....clle hhbbbllltt..heeryy...dddisrrrrrr undersaanlgeggnnaggoogii.oaoooee
Geometry
area calculation skills, geometry skills,
combinatorics wtw.iinttrrcs.annprroaaaaaaiaaie wtt.prrbbbbiiiyssiillll
coordinate.geomeen foorriatteeeettttr....m.rrrmmmmn
other www.ggeeetiiiillls.s...rmm poometrrricssillllyyypogrreeeeee pythogeeesttggrr..ss.lsll,,
matio_and_proportionn. skills,
trigonometry.skills, understanding circle properties and algebraic manipulation
Table List of skills used for annotating questions in each subject in MATH dataset
16

---

Published as a conference paper at COLM 2025
Subject
List of Skills
Per subject split in MATH
Intermediate Algebra
absolutee.valle_skill.
calculus..pptmmiaatioo.s..l.
function aonrrtto...oogsssttooooaaarararn
araahuuundesttaddddinnnnnntttttn inequality solving and understanding, polynomial skills,
properties and application of exponents,
nuadratic.equaaiionsaadd.dd
reuursiveffuuntttoosaaaaeeeeeee
solvinng
Salving..yystens equuaiion,
summation and onw..anllsiissofserreee.. oft.ssfffseeseeeeeeeeeeeeeeeeese
understtaniingggn...ppllyii.oono
understanding and utilizing infininte series,
understanding www.ppeeepppeeeptteii.eeeeeeeee. oedersswwiig...eenimnmooorrrrrre
Number Theory
ari urithhetticc_sequnnees. biso basic.arithmetic, divistion.and_rrmaanderrs,, exponentiation, factorization, greatest common divisor calculations, modular arithmetic, number number.manipulatte manipulation, number number.theory, polynomial operations
primmennumber_theod dequencceanalysiss solving.euuationss understanding.ooffrrae .fraciion
Pre-algebra
average calculations, basic arithmetic operations, circles, counting and number theory, exponentiation rules,
fractions_and_decinal geometry,
httpre/ssseettiiiaaiooaaaaaadiaa perimeter.andLared, probabiltty..nnd.commmmmntttosss ratioaaaaaaaaaadooprrootoooo solvingglinearreeuuaionn
Pre-calculus
algebra_.ad.eeuutiion,, basicc.rrigonomerryy calculus,
complexxnuubeer.ooerattiioo complex.numbers,
coordinate..systems, det.rmmaitttaaaaaccc....llledd
geometric.relations,, http:/inssngoommttranannssssseee calculation, ccometry_triang//e properteeeppppriisseee matrix wawrii.ooeeaattteeeeeeeee operations,
parametrin barametric.equations, sequences series and summation, three dimensional geometry, trigonometric calculations, vector pocctoroppeattins
Table 7: List of skills used for annotating questions in each subject of MATH dataset (continued from Table 6)
List of skills used for annotating questions in GSMSK
ieaeancedaarthmmetiiee algebra, basic.arithmetiin conversion.unit.oprraaa division, equations, fractions, interest calculations, multiplication, percentage, problem solving, profit loss, ratio proportion, time speed distance
17

---

Published as a conference paper at COLM 2025
A.2 Missing skill Identification from Model Responses
As described in Section 2.3, we use GPT-4o-mini to label the skills that are missing from model response. We ask the LLM to read the question along with the SLM response and provide the skills that the model fails to leverage in the response, from the given existing skill list. Below we show an example prompt for labeling missing skills for MATH Number Theory questions, as well as an example LLM output.
18

---

Published as a conference paper at COLM 2025
Example missing skill identification prompt for a MATH Number Theory question
[TASK]
You'll be given a math question and a step-by-step solution written by a Small
Language Model. Your task is to output:
(1) <judge > judge here whether the solution is correct or incorrect </judge >
(2) <reason > if it's incorrect, reason here why the solution is incorrect < /reasoo
(3) <skill > list here what skill(s) should the SLM enhance in order to answer correctly, seperated by commas </skill>> >.
[SKILL LISI]
You should only choose the skills from this list:
""arithmetic seaenneicsseuuncsss....
"base conversion",
"basic arithmetic",
"division_anc" remainders", "exponentiation", "factorization", greatest common divisor divisor.calculaat calculations", "modular arithmmtii,,,, "number manipulation",
"number theory",
Loperations", "polyromialoperraionnss" "polynomial
"rime nwmber nunber.theorrr"""" leeeuenceeanallyeeee solving equations", "understanding of fractions"]
[QUESTION] Given that p 2 is a prime number, evaluate
1 2 1 3
1- ....... +
1)-1
(mod p).
[MODEL SOLUTION]
To evaluate the sum 1 1 13 1
1, (p-2)
(p
(mod p), where p is prime number greater than or equal to 7, we start by noting that for any integer not divisible by p, the modular inverse a is the integer b such
that ab = (mod p). This means that
=1 (mod) p).
We can rewrite each term in the sum S follows: a-1. (a 1)-1 (a + 1)-a -1 (a 11 (a ++1)-a" -1 (mod p).
Now, let's consider the sum S more carefully. E a- (a 1) 2a (1) -1 (mod) p).
We can write: s
This becomes = 3 [1+1] = - 2) + Σ -1 It is known that F a1 0 (mod p), so IP- a1 =-pp 1) = -pp- - (mod p), since 1p-1) =1 (mod p). Therefore, S = (p 2--22 2) 1p-1) = -1 (mod p).
Thus, the value of the sum is
[REASON AND SKILL(S))
19

---

Published as a conference paper at COLM 2025
Example GPT-Ao-mini output for missing skill identification
(1) <judge> incorrect </judge>
(2) <reason>> The solution incorrectly simplifies the expression and misapplies prop- erties of modular arithmetic. The solution claims that the sum of the inverses from 1 to ( (p-1) is (-1 pmodp), which is incorrect. The correct approach involves recog- nizing that each term (a 1. (a + 1) 1) should be evaluated individually, and the should be calculated directly without assuming incorrect simplifications. The solution also fails to correctly apply the properties of modular inverses and does not properly evaluate the sum. </reason>
(3) <skill> modular arithmeticc number number.theory, understanding of fractions
</skill>
A.3 Skill-based Example Retrieval
We outline our algorithm for retrieving in-context examples tailored to a specific set of skills. Leveraging the Skill-Map definition in Section 2.1, which annotates each question with its associated skills, we construct an inverse mapping called Example-Bank Skill-Bank( ()) + P. This map associates each skill with the subset of in-context examples in the pool P that are linked to according to Skill-Map. Given a question and target skill set K, we retrieve in-context examples by randomly selecting one example from Haaple-Bankkff for each skill s in K. The algorithm is given in Algorithm 1.
Algorithm 1 Skill-based example retrieval
Input: List of skills K = [k1,.., kn (n < 5) Output: Selected 5-shot examples (et,
1: E <--0 []
2: if K is not empty then
3:
> We allow an additional repeated in-context example for the first n skills do E' <- Example-Bank (kg)
4:
for = 1 to
5:
6:
if E' is not empty then
7:
è - random_choice ( (E')
8
E+- [e]
9:
end if end for
11:
12:
for each in do
13:
E' +- ExxmmleeBBank()
14:
E' is not empty then
15:
e random choice (E')
16:
F + [e]
17:
end if
18:
end for
end if
20:
22:
E SE)
b> Remove repeated instances
if len then
24:
Append examples from fixed in-context examples to fill remaining shots
25:
> This happens in the rarest of cases when we don't have enough examples for
skill!
26: end
27:
return E
20

---

Published as a conference paper at COLM 2025
B Ablation Study
B.1 Ablations the reward filtering method in Stage 1
Recall that in Stage of the AdaptMI pipeline, use off-the-shelf process reward model (RLHFlow (aflow/Llama33 /L//amammm31180
to score small language models' responses, in order to filter out set of difficult questions for each model. Here, we conduct various ablation studies on the reward filtering process.
Out-of-distributio (OOD) prediction performance of reward model. Although we pri- marily evaluated AdaptMI on MATH and GSM8K, our method can potentially be extended to other math datasets. While the reward model we used in Stage was only trained on the MATH and GSM8K distribution, we show that capable of scoring responses for various OOD math datasets. Table 8 reports the reward model's performance on classifying correct/incorrec responses from Owen2.5-7B-Instruct on four popular math benchmarks: AMC23, AIME24, AIME25, and MATH The reward model achieves comparably high per- formance on scoring SLM responses these OOD, significantly more difficult benchmarks, indicating that the model is highly generalizable. This implies the potential to extend our method new datasets without the need train a specialized reward model for each one.
Metric
AMC23
AIME24
AIME25
MATH²
Accuracy
92.5
86.7
86.7
84.8
Precision
90.9
92.6
86.7
95.2
Recall
92.6
100.0
88.5
F1
93.0
92.6
92.9
91.0
Table 8: Reward model prediction metrics across four OOD math benchmarks. Despite not being trained on these benchmarks, the reward model's prediction capability is largely generalizable to them.
Reward Filtering vs. Simple Heuristics for classifying difficult questions. Considering the computational overhead of calling separate PRM, we explored alternative approaches to classifying questions that rely computation-free simple heuristics. Specifically, we experimented with two heuristic strategies:
Consistency heuristic: We measure the consistency of the model across five sampled generations per question and classify questions with lower consistency difficult. Specif- ically, a question is difficult if, among 5 sampled generations, the most common response appears < 2 times.
We use the length of the model's responses as a proxy and classify questions with longer responses as difficult. Specifically, a question is difficult if the average model response length on this question 2 800 words.
Length heuristic:
Table shows that both heuristics yield reasonably accurate predictions. Moreover, apply- ing AdaptMI on top of these heuristic-classffie difficult questions can improve the final accuracy by 2%. However, we leave a more thorough investigation into the robustness and generalizability of these strategies in relation to PRM-bbseed classification for future work.
Process Reward vs. Outcome Reward. We also compare the prediction accuracy of our process reward model (PRM) with threshold filtering (see Section 2.2) against directly load- ing the reward model as an outcome reward model (ORM). Our preliminary experiments indicated 0.9 the optimal threshold for the outcome rewards. With T = 0.9, the prediction metrics of the ORM are: Precision = 0.54 / Recall 0.90 / = 0.68, whereas the prediction metrics of the PRM with optimal thresholds are Precision 0.70 Recall 0.92 F1 0.80. Therefore, our method using PRM with threshold filtering is superior to directly using ORM.
21

---

Published as a conference paper at COLM 2025
Classification method
Classification accuracy
SLM accuracy w/o AdaptMI + w/ AdaptMI
Consisteney Heuristic
79.80%
52.8% 54.8% (+2.0%)
Length Heuristic
74.20%
52.8%
54.6% (+1.8%)
Reward Filtering
78.00%
52.8% + 56.4% (+3.6%)
Table 9: Performance of consistency heuristic and length heuristic on classifying difficult questions. The classification accuracy of simple heuristics are on par with the reward filtering method. Applying Stage 2 of AdaptMI on top the heuristic-classifed difficult questions can yield improvement on the final accuracy by 2%.
B.2 Comparing few-shot instructions with natural language instructions
Here, we explore an alternative strategy to construct adaptive in-context instruction. We want to test whether additional supervision from the LLM in AdaptMI+ could be provided in terms of feedback using natural language instructions.
modify our adaptive instruction as follows. After getting the predicted missing skills on model's response from an LLM, we prompt the LLM back with the missing skills and the corresponding skill-based in-context examples and ask the model to return concise natural language LLM feedbuck that contains criticism on the model's response, and hints on how to apply the required skills. See below for an example prompt.
For difficult questions,
MATH
GSM8K
Fixed Examples
74.7
91.7
LLM Feedback
74.4
91.2
+ Fixed Examples
74.5
91.1
Skill Examples
74.5
91.1
Table 10: Qwen2.5-7B-Instrruc accu- racy under LLM-generated natural language instructions.
We report the behavior of modified AdaptMI+ on Qwwn2.5-7BBInsttrue
Interestingly, we observe that even 7B models tend to not benefit from the unstructured instructions (see Table 10). Furthermore, even if skill-based in-context examples are utilized along with LLM feedback, the SLM's performance remains nearly unchanged, which suggests the model simply ignores in-context information that contains long, and unstructured natural language feedback.
22

---

Published as a conference paper at COLM 2025
Example feedback labeling prompt
[TASK]
You'll be given a math question, step-by-step solution written by a SLM, a step that is likely to be incorrect, the missing skills in the solution that you identified earlier, and the skill-rated in-context examples. Your task is to output:
(1) <comment>> comment here about the SLM solution < </comment>
(2) <hint> write here compactly the hints we should give the SLM to help it correctly answer this question next time. You should not include the question answer of this specific question. </hint>
[QUESTION]
Square ABCD has side length 2. A semicircle with diameter AB is constructed inside the square, and the tangent to the semicircle from C intersects side AD F. What is the length of CE? [Figure]
[SOLUTION]
To solve the problem, start by identifying the coordinates of the points in the square ABCD Let's place the square in the coordinate plane with A (0,0), B (2,0)
. (2,2), and D (0,2). The semicircle constructed with diameter \overline|AB] so its center is at the midpoint of (overline(AB], which is (1,0) and its radius is 1.
The equation of the semicircle (x-1)2 y for y 'geq 0 We need to find the point E where the tangent from C to the semicircle intersects side (overline(AD) (AD). its coordinates are (0, y y_E). The slope of the line The equation of the line CE y-E 2). \ Since = (0, y.E), substituting x = and y E into the equation, we get: V[y 2 |frac y (-2) y E AJ This confirms that the point E lies on the line. To find the exact coordinates of E, we use the fact that the tangent to the semicircle at point y-1) has the equation: M (x_1
Since lies on \overline
from C to E is ffracly_E- 2 2) =
is: y 2
- -1)(x 1) + 1 y = 1. \] Since the tangent passes through C (2,2), substituting 2 and y = into the tangent equation, we get: { (x_1 1)(2 1) yy ccdot 2 iimplies x_1 1 2y_1 = iimplies x_1 + 2y_1 = 2. \] Also, since (x_1, y-l) lies on the semicircle, we have: (xx
\ We solve these two equations simultaneously. From x x 1 gex 2y Substituting this into =
the semicircle equationn we get:
1-2=1 iimplies (1 (1-2y 1) +
y_1²2 = y-1(5y_1 - 4) = to the point A which is not valid tangent point, we have y_1 = \ffra( (5)
\implies 5y 4y_1 \implies ( Since y = corresponds
Then x.1 2- /cdot =5) =frrcc2)(5) The point of tangency is lleft(
(fac{2) ).fra((4(55 |right) The length of /verline/CCE) is the distance
/sqrt((200) \ from C= (2,2) to left((,, (frac(4 CE
\left(2
(frac[4) (ing55))rrght2)))) 2) (sqrt)4 (r) (rac ())
((iggh))))) right))2]
\sqrt(4
>left( ( (rac [arr (rcc
|srr( (,r
( frac((00) +
of (overline is (oxxe(((facc()ss(((34)33} (
Thus, the length
INCORRECT.STED
The slope of the line from C\) E\) |fracfy 20 \frac(2-y
The equation of the line CE)) is: \[ 2 /frac(2 (xE]22(x 2). \ Since E (0, y_E))), substituting 0%) and y= E.E into the equation, we get: \[y_E
(frac (2) 2. \i This confirms that the point E\) lies on the line.
[MISSING SKILIS]
siangleegeometrr-skkllls...iii gew.mltrreegeerrreeeeeooooo skgemmmetskkllllaggrraa......k
(COMMENTAND.HII
23

---

Published as a conference paper at COLM 2025
Example feedback from GPT-Ao-mini
<comment>>The The SLM's solution is mostly correct but could benefit from a more detailed explanation of the triangle inequality and how it affects the possible values of the third side. Additionally, the SLM might have overlooked the strict inequality in the triangle inequality theorem, leading to the incorrect conclusion that the minimum value of BC is exactly 2x </comment>
<hint>
Emphasize the importance of the triangle inequality theorem, specifically a a c, where a b and are the sides of triangle. Explain why BC must be strictly greater than 2x to satisfy the inequality BC +
3x.
Clarify that the smallest integer value for BC that satisfies this condition is 2x + +,, leading to the calculation of the maximum possible height h. </hint>
B.3 Fine-grained analysis of skill-based and fixed in-context examples on original manual split of MATH dataset
Fixed Examples
Skill based Examples
91.3
MATH Level
2
14.1
64.0
2
172
696
3-
76..
3
521
5
42.1
42.5
5-
842
567
20
40
60
80
300
400
500
600
700
800
900
Accuracy (%)
Average Output Length (Token)
Figure 5: Accuracy and average output length of Qwen2.5-33-Inssteeee on questions of Level 1-5 defined in the MATH dataset. Compared to Figure the performance gap between fixed and skill-based examples is unnoticeable across all levels.
We repeat experiment from Section 4.1.1. However, sampling to split the evaluation set into levels, we use the manual split of questions given the original MATH dataset. We report comparisons between skill-based and fixed in-context example selection strategies in Figure 5.
instead of using Best-of-n
Interestingly, the differences between the ICL performance and generation length with skill-based and fixed in-context examples for the SLM are less pronounced across the difficulty levels, compared the results in Figure 3. This suggests that the manual difficulty split in the MATH dataset may not align well with the model's own perception of question difficulty. To capture more fine-grained distinctions between the two strategies, using the model's s own responses through Best-of-n sampling serves as more reliable indicator of question difficulty.
C
Case Studies
In this section, we conduct case studies to gain deeper insight into how skill-based in- context examples might harm performance easy questions, mentioned in Section 4. We present two questions where SLM successfully solves with fixed examples, while failing with skill- -asse examples.
24

---

Published as a conference paper at COLM 2025
C.1 Skill-based examples lead the model to overlook key problem constraints
In the example below, the Qwen2.5-78-Instruct model is given an algebra question that includes multiple geometric constraints. While the question involves both Geometry and Algebra, it is only classified as an Algebra question in MATH, hence being combined with algebraic skill examples. When prompted with fixed examples, the model correctly identifies two possible answers and chooses the correct one according to the given condition "both coordinates are negative." " On the other hand, when conditioned by examples that represent algebraic skills, the model overly emphasizes algebraic completeness but overlooks this important problem condition. It finally selects the incorrect answer by random guess.
Case Study 1: An example skill-based prompt
|im start >>system
Please reason step by step, and put your final answer within \boxed (- lim_end
< iim_start |>user
[xxmmple with skill "distance cnnttance.and....e wwwdmmmidddsiiiiiiiiiiii skills"]
Example with skill "distance_and liteaanceeanddmiiie www.mmmidd.siiiiii.ommmm
|Example with skill "distance and midpoint skills"]
[Example 4 with skill "algebraic expression skills"]
[Example with skill "arithmeticcskills"]
A point y) on the coordinate plane with both coordinates negative is a distance of 6 units from the x-axis. It is distance of 15 units from the point (8,3). It is distance /n from the origin. What n??
|iimendd
lim start >>sssistatt
25

---

Published as a conference paper at COLM 2025
Case Study Correct model response under fixed example prompt
To solve the problem, we need to determine the coordinates (x, y) of the point that satisfies the given conditions and then find the distance from the origin.
Distance from the x-axis:
The point (x, y) is 6 units away from the x-axis. Since the y-coordinate is negative, we have:
| iimples y =6
Distance from the point ((88, 3)\):
The point (((, y) is 15 units away from MI \sqrt((x 8) + 33 = 15
Using the distance formula:
This gives us two possible values for ((})): x 8 12 = 20 'quaad [text(or) 'quaad = 8 12 = -4
3. Distance from the origin:
We need to find the distance from the origin \((0, 0)\) to the point ((x, y))) The distance formula is:
M \sqrt(x22 + y²2) V
We have two possible points: ((200,6))) and (((4,, (()))
For \(200, -6){):
/ \sqrt(2022 + (-6)-2} = \sqrt + 36) \sqrt(436) \]
For ((-4, (6)
M (sqrt((-4)22+ + (-6)2] 16 + 36) |sqrt(52) \J
Since the problem states both coordinates are negative, only \sqrt(52] is valid.
The correct value of (n\) is: 52
Thus, the final answer is: \boxed (52)
Case Study 1: Incorrect model response under skill-based example prompt
To solve the problem, we need to determine the coordinates (((, y) of the point that satisfies the given conditions and then find the distance from the origin. Distance from the ((x))-axis
[Same as correct response]
2. Distance from the point ((88, 3)1):
[Long equation-solving process]
3. Distance from the origin:
We need to find the distance from the origin \((0, 0)!) to the point \((x, y))). We have two possible points: ((20, -6)1) and ((-,,())).. ((20, -6)1; \/ ssqrt/202 + (n= 436 436 For (((4, -) | (sqr(((44)2 + (-6)2) |sqrt/16 = = = =2)sqrt(133 \J Thus, (nn
(-6)2) = bggrt/400+ + 36) (0sptt/8666=2issq((00))111)11
= -52)). Since the problem does not specify which point to use, we consider the first
solution where (n 43ee..fhe,,,,,,, the value of (n)) is \(1boxe(443)))) X
26

---

Published as a conference paper at COLM 2025
C.2 Symbol-heavy skill-based examples cause the model to overthink.
The question below requires plug-in-and-test approach instead of solving an equation. With fixed in-context examples, the model is able to find out the correct answer by directly plugging in and trying out small values. However, the skill-based examples that involve equation solving may have caused the model to overthink. After failing in the first plug-in- and-test, ended up attempting to solve the equation system and eventually failed.
Case Study An example skill-based prompt
startt >>system
Please reason step by step, and put your final answer within \boxed |imm_nd >>
[im start |>user
[Example with skill "sequence and series skillss
Example with skill "sequence and andiesrres... series.skills")
|Example with skill sicquence_and_seee soddsseree_ssille siilisssssills]
[Bxample with skill "function conpptioocoomooitioin skills"]
[Example 5 with skill "solving Sadviingequations"]
Charlize accidentally omitted two consecutive integers when adding the elements of the arithmetic sequence, (1,2,3,...,,t) n}. smallest possible value of n??
the sum she obtained is 241, what is the
< lim_end >>
im_start
>assistant
Case Study 2: Correct model response under fixed example prompt
To solve the problem, first need to determine the of the arithmetic sequence ((1(1, 2, 3, Idots,, n)).). The sum of the first ((n)) positive integers is given by the formula: S [rac(((n+1)) A]
Let the two consecutive integers that Charlize omitted be \(k)) and ((++))) The sum of these two integers is: M k (k+1) 2k + - \ Therefore, the sum of the sequence with these two integers omitted is: S (2k 1) (f)) (2k
+ +1) J We know that this sum is 241, we can write: {[r
(2k + 1) = 241 \] Rearranging this equation, we get: \[ (2) = 242 + 2k AJ Multiplying both sides by 2, obtain: M[ n(n+1) 484 4k \) Since \(k)) is integer, \(884 + 4k)) must be a product two consecutive integers. We need find the smallest (n)) such that ((((n+1))) is close to 484 and is of the form \(844 4k)).
Let's check the values of \ (n)) starting from the smallest possible (nn) that is greater than \()sqrt( (84] = 22)):
If (n = 22)), then (n(n+1) = 22 timee 23 = 506)). We need to check if (506 = 484 + +4k)): 506 484 22 (implies 4k 22 (implies k \frac (4) 5.5 Since
\(k)) must be an integer, (nn 22)) is not solution.
If (n = 23)), then (n(n++)) = 23 tiims 24 552)). We need to check if (552 484 17 Since ik 17() is an integer, (n= 23)) is solution. The two consecutive integers omitted
+ 4k)): \[ 5522 484 =8 (implies 4k = 68 \implies k |frac(68)
17 and 18. Therefore, the smallest possible value of (n)) is: \/ (boxed(23) \
Thus, the final answer is: bbxedd(2)
27

---

Published as a conference paper at COLM 2025
Case Study Incorrect model response under skill-based example prompt
To solve the problem, we first need to determine the sum of the arithmetic sequence /( (I, 2, 3, Idots, } )). The sum of the first (nn \) positive integers is given by the formula: \[S [rac([(n+11){2)) V
Charlize omitted two consecutive integers, say \(k \ and (k+1 1). Therefore, the
sum she obtained is: V (k + (k+1)) (2k 1)= (fan<///))////////n/n/aaaa- (2) (2k 1) \I 241 | Rearranging the equation, we get: \I ( 1 = 2k \ Multiplying both sides by we obtain: \[ n[n+1) 486 +4 Since \(k \)
We know that this sum is 241,
have: \[ (facc(((++()))))) (2k
integer, (486 + 4k \) must be a product of two consecutive integers. We need to find the smallest (n \) such that (n(n++1) )) is close 486 and is of the form \ + 4k \).
Let's check the values of \(n)) starting from the smallest possible ((n)) that makes \n(n+11 > 486)): For (nn 22)): ): [[n(+1) = 22 ttmes 23 = 506 \J We need to check (5066) can be written as (486 4k(): \[ 506 486 4k \implies 4k 20 \implies == \] So, \n=22) works with (kk =
5 The two consecutive integers omitted are and Let's verify the sum: The sum of the sequence \(M(1, 2,3, |dots, H1 is: \I fraac (22 times 23 } {2 253 \J If we omit 5 and 6, the sum is: \ 253 (5 6) 253
242 \ This is not correct.
Instead, let's try to solve the equation:
[Model attempts the solve the equation]
[Model reaches maximum output length before finishing] X
D Additional Results
D.1 Classification results of easy and difficult questions
In Stage of AdaptMI (see Section 2.2), we identify a set difficult questions for each individual model using process reward model along with filtering heuristic. Table 11 reports the proportions domain. Compared to Table 1, the proportions of difficult questions closely correspond to the accuracy numbers of each model, even though we did not access the ground truth in the whole pipeline. Notably, our classification method captures not only questions that the model gets wrong, but also questions that the model passes with a flawed solution process.
difficult questions classified for different models in each math
Model
Geometry
Precalculus
Algebra
Prealgebra
Intermediate Algebra
Owenn255-1.0-Innttruct
1.---ruutttt
74.9
45.0
82,2
61.8
70.1
29.7
33.2
75.9
Qwen2.5-78-Instteee
59.3
67.9
29.1
29.3
72,9
1lama-32-11--nnsmma
93.5
92.0
91.4
89.7
99.0
1llmma3.2-30-Inssttee
68.2
82.7
45.5
48.9
85,7
Model
Pront.&Prob.
Number Theory
MATH Avg.
GSMSK
Qwen2.5-11 1...2.5-1.BB-nsstruce
70.3
65.2
61.9
48.6
Sween.553B--nssheee
62.2
56.1
52.1
26.6
Owen2.5-7B-Instteee
49.5
1.lama-3.2-18 21---ruuttt
97.9
95.2
94,0
72.8
1lama-3.2-30-Innsll
62.3
Table 11: Proportions of difficult questions (%) classified by AdaptMI for each model. Although our method did not access the ground truth, the proportion of classified difficult questions still closely mirrors each model's accuracy (see Table 1) in each domain.
28

---

Published as a conference paper at COLM 2025
D.2 AdaptMI and AdaptMI+ performances
In addition to Table 1, we put the accuracy results on Number Theory, Intermediate Algebra, and Counting & Probability in MATH in Table 12. These results align with each other- AdaptMI and AdaptMI+ yield substantial improvement compared with all Pass@1 baseline, while being on par with the Consistency@5 results.
D.3 Effect of skill-based examples on difficult and easy questions
In Section introduce observation that skill-based examples only boost SLM performances on difficult questions but harm performance on easier ones. We present the additional results on Qwen2.5-3B-Instruct and Qwent.5-7B-Instruct in Table 13 and Table 14. Similar to Table 2, there is a clear performance drop on easy questions with skill-based examples, although the drop for Qwen2.5-3B-Instrus and Qwen2.5-7B-Instruct is less significant than S.wen..5-1.B--Insite
29

---

Published as a conference paper at COLM 2025
Number Theory
Intermediate Algebra
Counting Probability
Methods
# Quen2.5-1.58-Instree
Fixed Examples
36.5
47.3
Random Examples
43.7
35.1
47.3
Skill-based Examples
45.4
35.8
44.7
Consistency@5
50.0
39.8
47.8
AdaptMI
49.8
36.9
50.0
AdaptMI+
49.1
38.4
51.5
# Qween2.5-33 5-B I--Inrucc
Fixed Examples
65.9
46.8
59.5
Random Examples
64.1
46.7
60.1
Skill-based Examples
66.1
45.9
60.3
Consistency@5
66.5
49.4
61.7
AdaptMI
66.7
46.5
60.6
AdaptMI+
68.9
49.8
62.7
# Qwen2.5-7B Instruct
Fixed Examples
74.8
57.3
72.6
Random Examples
74.4
55.7
73.4
Skill-based Examples
73.0
55.9
71.1
Consistency@5
57.5
71.7
AdaptMI
73.5
57.6
71.5
AdaptMI+
77.4
58.8
74.9
# +Llama-322-1 1--ssruuttt
Fixed Examples
7.8
11.5
Random Examples
7.3
7.9
6.9
Skill-based Examples
11.2
7.3
10.4
Consistency@5
21.4
6.7
14.3
AdaptMI
12.1
7.8
12.5
AdaptMI+
10.3
8.9
13.5
# LLaama-3.2-88 3--ssr------ Instruct
Fixed Examples
38.7
22.6
42.7
Random Examples
29.3
18.5
33.7
Skill-based Examples
39.6
23.2
33.7
Consistency@0
35.0
21.1
46.7
AdaptMI
24.4
39.3
AdaptMI+
24.4
39.3
Table 12: Additional results of Table 1. AdaptMI and AdaptMI+ also demonstrate consistent accuracy gain compared with baseline methods. All results are Pass@1 accuracy unless otherwise indicated. Exp. stands for Examples. The selection methods for fixed, random, and skill-based examples are introduced in Section 2.1
30

---

Published as a conference paper at COLM 2025
Question & Example
MATH
Geometry
Precalculus
Algebra
Prealgebra
Number Theory
1.omn.55-15B-Imstrra
Fixed
21.3
23.7
44.8
35.1
24.1
Random
23.2
25.3
53.9
40.5
21.9
Diff.
28.4
28.9
55.1
45.5
31,2
Skill
+7.1
+5.2
+10.3
+10.4
+7.1
Fixed
82.1
81.8
94.6
93.7
84.6
Random
81.6
78.9
92.1
92.3
80.1
Easy
77.2
71.5
85.9
86.0
71.8
Skill
-4.9
-10.3
-8.7
-7.7
-12.8
Queen2.53B-Instrucc
Fixed
36.5
60.6
49.5
Random
36.8
38.7
62.6
50.5
49.2
Diff.
34.1
41.8
68.3
54.3
50.8
Skill
-2.4
+3.9
+7.7
+6.2
+1.3
Fixed
88.5
90.2
95.9
95.4
86.9
Random
83.6
86.5
94.0
94.0
84.1
Easy
84.7
88.3
93.8
93.8
85.7
Skill
-3.8
-1.8
-2.2
-1.6
-1.3
Qween.557BBInstruu
Fixed
50.0
51.3
80.1
71.6
66.8
Random
48.3
52.5
81.3
71.3
67.3
Diff.
52.0
57.4
81.5
74.7
66.9
Skill
+2
+6.1
+1.4
+3.1
+0.1
Fixed
90.8
93.9
98.7
97.5
93.3
Random
92.6
99.2
97.7
91.9
Easy
89.8
91.4
96.0
94.7
91.5
Skill
-1.0
-2.5
-2.7
-2.8
-1.8
Table Accuracy Qwen2.5-1.5B. Instrucc,, Srren2.5-3B-Instuuc and Qwen2.5-7B-Insttreee on difficult and easy questions, respectively under fixed, random, and skill-based examples (additional results for Table 2). Skill-based examples boost performance on difficult questions across all categories, while significantly underperforming on easy questions. The gap between easy and difficult questions is more pronounced for smaller models.
31

---

Published as a conference paper at COLM 2025
Question & Example
MATH Counting & Probability
GSM8K
Intermediate Algebra
Avg.
Avg.
Qwen2.5-1.550 5-.52.5551-BBtssrrrr
Fixed
27.0
28.2
29.8
45.2
Diff. Random
23.0
27.2
31.2
46.1
27.4
32.1
35.7
48.0
Skill
+0.4
43.9
+5.9
+2.8
Fixed
80.7
92.2
90.2
96.3
Random
75.7
88.1
87.6
90.6
Easy
74.5
74.5
81.0
83.2
Skill
-6.2
-17.7
-9.2
-13.1
Fixed
35.5
40.3
51.6
Random
36.2
41.0
43.4
56.7
Diff.
35.0
42.0
45.2
61.8
Skill
-0.4
+1.69
+2.3
+10.2
Fixed
82.6
91.1
92.4
96.7
Random
81.8
90.9
91.8
95.6
Easy
79.8
90.5
90.4
93.9
Skill
-2.8
-0.6
-2.0
-2.8
Fixed
50.0
61.5
60.7
74.1
Random
49.1
62.7
60.8
76.7
Diff.
51.2
61.9
62.7
77.0
Skill
+1.2
+0.4
+2
+2.9
Fixed
89.7
96.1
96.2
97.3
Random
86.5
95.3
96.4
Easy
86.1
94.7
94.1
95.5
Skill
-3.6
-1.4
-2.1
-1.8
Table 14: Accuracy Qwen2.5-1.5B. InIstuuu,,, Swee2.5-3B-Instruu and Qwen2.5-7B-Insstee on difficult and easy questions, respectively under fixed, random, and skill-based examples (additional results for Table 2). Skill-based examples boost performance on difficult questions across all categories, while significantly underperforming on easy questions. The gap between easy and difficult questions is more pronounced for smaller models.
32


Reviews:
[{"review_id": "z5AAdqOvjE", "reviewer": "Reviewer_8zxM", "summary": "The paper introduces two methods AdaptMI and AdaptMI+ for selection of in-context skill based examples to promote in context learning in small language models for math datasets. They specifically observe that vanilla skill based examples hurt small language models on easy questions due to unnecessary information and hence require carefully chosen examples. Their methods show significant improvements in accuracy in some areas like pre-algebra.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ The paper provides a simple yet powerful method for selection of relevant examples for in-context learning. \n+ They make clear observations on the performance of small language models on fixed set of examples and skill based examples", "reasons_to_reject": "+ The authors do not provide details of the reward model used\n+ The authors have also not provided a skill wise breakdown of performance of SLM: for example:  Which skill-based examples resulted in most improvement in accuracy?", "questions_to_authors": "+ Why does skill based examples hurt SLM performance on easy questions? Can the \"unnecessary information\" or \"cognitive overload\" quantified ?\n+ Is there anything specific to pre-training of SLM that affects the performance of in-context skill based examples.\n+ Looks like most gains are made in pre-algebra and the model struggles to make significant gains in geometry ? Why is that ?", "extra_scores": {}}, {"review_id": "Q5kFTMzGIR", "reviewer": "Reviewer_HGMM", "summary": "The paper mainly explores strategies of selecting examples for in-context learning (ICL), to improve the performance of smaller language models (SLMs, around 1B-7B parameters) in solving mathematical problems. Since different questions require different sets of *skills*, one possible strategy is to select examples that demonstrate related skills. Counterintuitively, this skill-based selection leads to worse performance for SLMs. The ablation study shows that the skill-based examples primarily benefit more difficult questions, but cause confusion for easier ones, possibly due to overthinking. The paper also proposes an adaptive method (AdaptiveMI) to address this issue. First, an external reward model judges the performance of the SLM, and opts in skill-based ICL examples only for challenging questions. Alternatively, a reflective LLM enhances the selection by picking skills the SLM fails to demonstrate (AdaptiveMI+). These methods improve performance to Consistency@5 level.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I agree with the authors’ point on L264 that “the in-context learning dynamics of SLMs are understudied”. By intuition, the skill-based selection can be interpreted as using the most relevant “task” for ICL, which should outperform showing less relevant or random “tasks”. But the results in this paper clearly indicates that such examples can negatively impact the performance on questions that are already easy. This is an interesting insight to the broader community.\n\nBesides, I think the paper is clearly written and easy to follow. One specific point I like is that instead of reusing the difficulty label within the MATH dataset, the experiments in Section 4 use difficulty labels derived from the SLM’s real performance. Difficulties judged by humans do not necessarily correspond with LM performance, yet the community seems to be less aware of this, so I feel this choice of difficulty metrics is also valuable.", "reasons_to_reject": "## Method Effectiveness\n\nThe AdaptiveMI method uses much larger models (for AdaptiveMI+, GPT-4o-mini is used, which is probably an LLM already) to enhance SLMs under skill-based ICL example selection. Surely this method improves the performance to Consistency@5 level, but isn’t it requiring more computation than Consistency@5, because of all the extra larger models? Therefore, it might be less economic as Consistency@5 in real scenarios, especially for AdaptiveMI+. Another bottleneck of this method is the SkillBank, whose construction seems to be impossible with SLMs only.\n\n## Additional Experiments\n\nPutting effectiveness aside, I think there could have been more experiments to better explain the negative effect of skill-based ICL. Since the difficulty of a question is based on LM’s performance, statistically analyzing the correspondence of the difficulty and the performance impact of the skill-based ICL should be possible. Also it might be a good addition to show the distribution of difficulty level within each MATH subtask.", "questions_to_authors": "## The overthinking hypothesis\n\nThe examples demonstrated to the SLMs are probably beyond their reasoning capability, therefore the ICL encourages overthinking. What about adding some statistics about these examples and the (over)thinking they triggered? In addition to Fig 3, how long are these examples, compared to outputs of different difficulty level? Further, what do you think of an ablation using ICL examples rewritten to each SLM’s own “reasoning style”?", "extra_scores": {}}, {"review_id": "sKZtXTcra5", "reviewer": "Reviewer_qoNx", "summary": "This paper uses an adaptive strategy to improve the efficacy of in-context learning (ICL) in SLMs, which are widely-known not to generalize well when given few-shot examples, specifically in the context of maths problems. This adaptive strategy relies on “skill-based prompting”, involving an intentional selection of ICL examples based on the difficulty of the question and the skill required to solve the question, to aid the model in improving its output.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Clear organization of the different sections; the paper flows well and is understandable to the reader\n\n- Good methodological considerations on what works best to optimize performance and why (through ablation studies)\n\n- Identified important gap in the literature on SLM and extrapolated their findings in the specific context of maths problems to a larger scope, pushing for adaptive learning strategies\n\n- AdaptMI and use of skill-based problems in ICL is well-justified in the discussion, and further supported by the fine-grained analysis by difficulty", "reasons_to_reject": "- Language use is anthropomorphic in some instances (e.g., “more likely to overthink”), which comes off as inaccurate, and the reference to cognitive theories as motivation for the learning strategy is bit surface-level and could be detailed further\n\n- Section 3.2: only analyzes instances where AdaptMI and AdaptMI+ do better, could expand further on specific cases where consistency@5 does better/is on par with their approach and why to use one over the other (e.g., efficiency)\n\n- Although skill-based prompting is properly justified for difficult questions, it is not clear that fixed examples are necessarily optimal for easy questions as performance is usually on par with randomly-selected examples.", "questions_to_authors": "- In the partition of easy and difficult questions, you use $\\tau_2$ as a lower bound of the reward value of any step. However, there could exist a case where the reward in one step could be low, but the average reward of the question is high (implying it is easy).  What would happen if we ablated $\\tau_2$ entirely? How does it affect the model’s overall performance? Would like to see more variation in the threshold values to observe this", "extra_scores": {}}, {"review_id": "wmUe2WelJS", "reviewer": "Reviewer_Qc6H", "summary": "In this paper, the authors propose AdaptMI and an improved version AdaptMI+ for adaptive skill-based example selection in in-context learning. The methods contain two-stage: first detect the difficult and easy questions based on rewarding the inference LLM’s response; second select examples that correspond to the question skills (for AdaptMI) or missed skills (for AdaptMI+). Experimental results on several Small Language Models (SLM) demonstrate that these methods outperform existing baselines. Moreover, the authors provide detailed discussions to validate that SLM is likely to “overthinking” on easy questions given skills.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is well written, with a clear and logical presentation of both the proposed methods and experimental setup. \n2. The authors provide a thorough and insightful analysis of how skill-based in-context examples affect the performance of SLMs. Their observations (e.g., such examples can introduce unnecessary cognitive load on easy problems) are well supported by empirical results.", "reasons_to_reject": "1. The paper contains some overclaims. For instance, in the Introduction, the authors claim that AdaptMI+ \"creates examples\" for the missing skills. However, in implementation, they simply select examples from an existing annotated pool based on those missing skills—similar to how skill-based examples are selected for all predicted skills. The term “create” misleadingly implies the generation or synthesis of new content for the missing skills, which is not supported by the implementation details.\n2. Certain technical details lack clarity (please refer to Q1 below), and there are notational inconsistencies, such as the variable $k$ representing both the number of required skills per question in Section 2.1 and the number of steps in a solution in Section 2.2. \n3. The overall efficiency of the proposed methods is limited. They rely on training an additional process reward model and using a large LLM to predict the SLM’s missing skills, which introduces high computational overhead and potential error accumulation. \n4. The logic behind AdaptMI+ is somewhat questionable. When using a large LLM to predict the missing skills, it implicitly assumes or requires that the large LLM can accurately solve the problem itself. However, if the large model can already solve the problem correctly, then AdaptMI+ is essentially providing the SLM with information that reflects the correct solution. It is akin to having a stronger model solve the problem first and then offering its reasoning path as a reference to the SLM. This makes me feel that the improved performance of AdaptMI+ is to be expected, but not due to a technical advancement, rather because it is indirectly leveraging the reasoning results of the larger model.\n5. Some important experimental results require further explanation and visualization (please refer to Q2-4 below).", "questions_to_authors": "1. In the construction of the Skill-Map, each question is associated with exactly $k$ skills. How do the authors ensure that every question indeed requires exactly $k$ distinct skills? \n2. Since the proposed methods heavily rely on the predictions of a large LLM (GPT-4o-mini) for both skill annotation and missing skill identification, it would be important to understand how reliable these predictions are. Could the authors provide a quantitative analysis of the skill prediction accuracy, ideally benchmarked against human-annotated ground truth?\n3. In Section 3.2, the authors state: \"This may suggest that higher-performing models require a more intelligent and target skill identification process.\" I am a little confused, because the same skill identification LLM (GPT-4o-mini) is applied to all model backbones regardless of their performance level. How can the experimental results support the claim that higher-performing models need more refined skill selection?\n4. Given that the core idea of AdaptMI/AdaptMI+ is to treat easy and difficult questions differently, it would be helpful to explicitly report the proportions of easy versus difficult questions as determined by the methods on each dataset.", "extra_scores": {}}, {"review_id": "fPjNwGXiN4", "reviewer": "Reviewer_nCJe", "summary": "This paper aims to improve the in-context learning (ICL) performance of Small Language Models (SLMs). It proposes AdaptMI, a method for selecting skill-based in-context Math Instructions for SLMs. An enhanced approach, AdaptMI+ is further proposed to identify the key skills missing in the SLM’s responses for difficult questions. The evaluation is conducted using the MATH and GSM8K datasets. The results show that both AdaptMI and AdaptMI+ methods outperform the baseline methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The structure of the paper is clear. The study presents a detailed experimental design and a thorough analysis of the findings.", "reasons_to_reject": "Null.", "questions_to_authors": "Concerning the evaluation presented in Table 2 and Figure 3, could the authors elaborate on the rationale for not including a comparison between methods AdaptMI and AdaptMI+ and the LLM?\n\nFor the results shown in Table 1, the AdaptMI+ method did not consistently outperform AdaptMI. An explanation or analysis of the factors contributing to these instances would be appreciated.", "extra_scores": {"ethics_concerns_details": "Null"}}]

Author rebuttal:

Meta-review: Pros: 
- This paper presents a technically strong method and its variation, AdaptMI and AdaptMI+, that significantly improves in-context learning accuracy in small language models. 
- Systematic analysis has been offered.

All reviewers agree this paper is above the bar of being presented at COLM. 

Cons that have been addressed in rebuttal and should be included in revision: 
- Additional analysis (break-down analysis, ablation on $\tau_2$), as listed in the rebuttal
- If space allows, please also consider adding some future work discussions to include content that the authors are not exactly sure during the rebuttal, e.g., 
> “Why are most gains in pre-algebra and the model struggles to make significant gains in geometry?”
Response: This likely reflects differences in pre-training; models may have seen more pre-algebra content than geometry. However, without detailed knowledge of the pre-training data, the exact cause is difficult to determine.

Cons that should be addressed in revision:
- Please reduce the use of anthropomorphic languages. I am in agreement with Reviewer qoNx, while I acknowledge that might have somehow become a community convention, it is always encouraged to avoid potential misleading to the public. And that's our responsibility as scientists.

[Automatically added comment] At least one review was discounted during the decision process due to quality]
Decision: Accept