Distilling LLM Agent into Small Models with Retrieval and Code Tools
Authors: Minki Kang
Jongwon Jeong
Seanie Lee
Jaewoong Cho
Sung Ju Hwang
Venue: NeurIPS 2025 spotlight ()
Keywords: langauge model
language agent
reasoning
distillation
Primary area: deep_learning

Abstract: Large language models (LLMs) excel at complex reasoning tasks but remain computationally expensive, limiting their practical deployment.
To address this, recent works have focused on distilling reasoning capabilities into smaller language models (sLMs) using chain-of-thought (CoT) traces from teacher LLMs.
However, this approach struggles in scenarios requiring rare factual knowledge or precise computation, where sLMs often hallucinate due to limited capability.
In this work, we propose Agent Distillation, a framework for transferring not only reasoning capability but full task-solving behavior from LLM-based agents into sLMs with retrieval and code tools. 
We improve agent distillation along two complementary axes: (1) we introduce a prompting method called first-thought prefix to enhance the quality of teacher-generated trajectories; 
and (2) we propose a self-consistent action generation for improving test-time robustness of small agents.
We evaluate our method on eight reasoning tasks across factual and mathematical domains, covering both in-domain and out-of-domain generalization.
Our results show that sLMs as small as 0.5B, 1.5B, 3B parameters can achieve performance competitive with next-tier larger 1.5B, 3B, 7B models fine-tuned using CoT distillation, demonstrating the potential of agent distillation for building practical, tool-using small agents.

Distilling LLM Agent into Small Models with Retrieval and Code Tools
Minki Kangl" Jongwon Jeong?
Seanie Lee'
Jaewoong Cho
Sung Ju Hwang 144
KAISTT University of Wisconsin-Madison KRAFTON, ADeepAuto.ai
fminkikang, sjhwang62)@kaist. ac kr
Abstract
Large language models (LLMs) excel at complex reasoning tasks but remain computationally expensive, limiting their practical deployment. To address this, recent works have focused on distilling reasoning capabilities into smaller language models (sLMs) using thain-of-thought (CoT) traces from teacher LLMs. However, this approach struggles in scenarios requiring rare factual knowledge or precise computation, where sLMMs often hallucinate due to limited capability. In this work, we propose Agent Distillation, a framework for transferring not only reasoning capability but full task-solving behavior from LLM-based agents into sLMs with retrieval and code tools. We improve agent distillation along two complementary axes: (I) introduce prompting method called first-thought prefix to enhance the quality of teacher- teacher-generated trajectories; and (2) we propose self-consisteni action generation for improving test-time robustness of small agents. We evaluate our method on eight reasoning tasks across factual and mathematical domains, covering both in-domain and out-of-domain generalization. Our results show that sLMs small as 0.5B, 1.5B, 3B parameters can achieve performance competitive with next- tier larger 1,5B, 3B, 7B models fine-tuned using CoT distillation, demonstrating the potential of agent distillation for building practical, tool-using small agents. Our code is available at https://github..oo. www.uuu.coo//aaaaaaaa wwoom//arriin/gggnttiiiiilllllli
CoT Prompting
COT Distillation ZIZ
CoT Distillation RAG
50
Agent Prompting
Agent Distillation (Ouns)
40
39.5
Accuracy (%)
30
2377
20
15.9
0.5B
1.5B
3B
7B
32B
(teacher)
Model Size
Figure 1: Performance comparison of different sizes of Qwen2. 1- S-Inttruct models [1] on the average accuracy of four factual reasoning tasks (HotpotQA [2]. Bamboogle [3], MuSiQue [4], 2WikiMultHooQAA [5] and four mathematical reasoning tasks (MATH [6]. GSM-HHrd [7]. AIME [8], OlymMATH [9]). Distillation done using the 32B model as the teacher and models ranging from 0.5B to 7B as students. Agent distillation consistently improves the performance of smaller models across hoth domains by enabling them to perform code execution and retrieve information for tasks adaptively. Full results are provided in Table 2.
WWor done at KRAFTON.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Train query: What would $100 invested Apple stock in 2010 be worth by 2020?
Chain-ofTThour Distillation
Agent Distillation (Ours)
Initial incestment:
Thought I will search Agole's histerscal stuck data froms 2010 2020, which includes both paice spúit events.
s100 pple stock at start 2010
AAPI price in Jan 2010 $30 (split- usted)
Action
soneniquaryy Spple price milewery
oo catsner)
Teacher (CoT)
Number of shores purchased:
Teacher (Agent)
s100 $30 3.35 shores
Observ.
Jan 2010 pice (split- sdjule-djuusel: $30, Dec 2020 price factual): s132 Spills: 7-for-1 onne 2014, 4-for1 on Auy 2020
Stuck splits during 2010-22000 2020:
2014: 7-fee- split - 3.53 x 23.33 stares
Thought
I'll get the price in 2010 and 2020, apply spliz, and computi Ennal value.
Distill
Price 2020. AAPL
20200: for split 28.33 x = 90.33 shares
Disiil
Action
pricce 2010 30; price_2808 132
Finsl
$152 $12,320
solics - [2, 43; shares intcial iives trent / price 2010
for salit in splits: shares the split #inel valuee shanes price 2898; print(final vullue)
Observ. 12,300
Test query: What would $300 invested in Nvidia stock in 2015 be worth by 2020?
Generalization with new search query
Initial investment $300 in NVIIA stock the of 2015
Thought
search for Nvidia's histonical stock from 2020, which includes both price and splitevents.
Student (CoT)
NVIIM price in =20
Student (Ages0
Action
sea Seancr(seery- "widia stock price history arts to sece inchuding sallitsr)
Number shares purchased:
$300 + $20 10 stares
Observ
Price Jan 2015 - $20, Price
$130
Stock splits during 2015-20000 shares
No stock split between 2015 and 2020
2019: 6-Eoe-1 splat
Price at end of 2020: NNDA is $130 per share Final value 40 x $130 -0:00-55 $3.200
Thought
can now compute how many shares $300 would buy 2015 and their value in 20120.
Action
initial, investment 200; price 2015 20; price
price_a020 2000 139
shares initial. Lnveestmnnt / price 2015
Hallucinations in wrong fact and wrong calculation!
paluue
petce_2828; peint(ftnn) vallve)
Observ. 1,50
Write code for new calculation
Figure 2: Concept. Chaiu-of-Thought (CoT) distillation trains student models to mimic static reasoning traces from LLMs, but often fails when new knowledge or precise computation is needed at test time. Our proposed agent distillation instead teaches student models to think and act (c.g., retrieve facts or execute code) offering stronger generalization and better robustness to hallucination.
1 Introduction
Large language models (LLMs) have achieved remarkable performance across complex tasks, surpassing average human accuracy on college-level mathematics and demonstrating com- petence in high-stakes domains [10-12]. However, as LLM usage grows, their high inference cost becomes increasingly burdensome. While these considerations have motivated growing interest in smaller language models (sLMs) [13, 14]. preserving the problem-solving capabilities of larger models in sLMs remains challenging. Therefore, a core research question emerges: how can we preserve LLM-level problem-solving ability in much smaller models?
Although recent advancements pre- and post-training methods have steadily increased the capabili- ties of sLMs [15], sLMs still struggle to solve complex tasks at the level of LLMs. To address this, recent works have explored reasoning distillation, where sLMs are trained to mimic CoT reasoning traces generated by teacher LLMs through next-token prediction [10, 11, 1, 16, 17].
However, distilled small models are prone to hallucination and often fail to perform accurate cal- culations [18]. For example, answering the real-world question, "What would $100 invested in Apple stock in 2010 be worth by 2020?". requires both factual knowledge about stock history and arithmetic reasoning. As illustrated in Figure 2, LLMs can correctly answer this question using CoT by leveraging memorized knowledge and numerical skills. However, simply distilling such reasoning trace into an sLM does not guarantee generalization, especially those involving new knowledge or calculation not observed during distillation due to their limited capability [19].
In this work, we propose Agent Distillation, framework that moves beyond static reasoning to distill the ability to take action with tools, from LLM agents (e.g., ReAct [20], CodeAct [21]) into sLMs through reason-act-ee acbee-es---eeeeeeeeee trajectories. Our goal is to equip sLMs with agentic capabilities: reasoning through problems, taking actions to use code or retrieval tools, observing outcomes, and refining their approach- approach-cloning the behavior of LLM agents. This approach offers two key advantages: (1) sLMs focus on learning how to reason and act to solve problems using tools, rather than memorizing knowledge and calculations, and (2) they generalize better to new queries requiring previously unseen facts or calculations. A remaining challenge is whether such complex agentic behavior can be distilled from large teacher model (>30B) into much smaller student (0.5-3B) [1].

---

To this end, we introduce two simple but effective methods to aid effective distillation. First, we propose a first- trst-thoughh prefix method that aligns agentic reasoning with the teacher model's instruction- tuned behavior, improving trajectory quality of teacher agent without additional fine-tuning. These improved trajectories offer better supervision for sI.M distillation. Second, we improve student robustness at test-time through self-consistent action generation, which samples multiple trajectories and selects the one yielding a valid and consistent outcome leveraging code interpreter.
We evaluate our agent distillation on four factual (e.g., HotPoiQA [2]) and four mathematical (e.g., MATH [6]) reasoning benchmarks. For each reasoning type, we consider one in-domain task and three out-of-domain tasks to test generalization. As in Figure 1, our results show that agent distillation consistently enhances the problem problem-solving capabilities of small models of 0.5B 7B.
To summarize, our work makes the following key contributions:
We propose Agent Distillation, a framework for training sLMs to imitate trajectories from LLM agents, enabling agentic behavior without memorizing factual knowledge and calculations.
We introduce two methods to overcome limitations of naive distillation: (I) first-thought prefix for improving teacher trajectory, and (2) self- celf-consistent action generation to boost test-time robustness.
We validate our method across 8 factual and mathematical reasoning benchmarks, showing strong performance across domains and student model scales (0.5B-7B) compared to CoT distillation.
Remarkably, demonstrate that even 0.5B, 1.5B, and 3B models distilled with method can achieve comparable performance to next-tier larger models distilled with CoT on average.
2
Related works
2.1 Reasoning distillation of language models
Large language models (LLMs) have shown strong performance on complex reasoning tasks using methods like chain-of-thougee (CoT) prompting [22, 23J. To transfer these capabilities to smaller mod- els (sLMs), CoT distillation methods [16, 17, 24-27] train sLMs to reproduce step-by-step reasoning traces from stronger LLMs. This has proven effective particularly in mathematical reasoning and now a common component of post-training pipelines [11, 1]. To improve generalization, recent methods incorporate external tools such as retrieval [19, 28, 29] or code execution [30-32], helping sLMs focus on transferable reasoning strategies rather than memorization of others. Still, most existing approaches rely on static demonstrations and lack interaction with the environment.
In contrast, we distill agentic behaviors where models learn the reasoning and tool use during interactions with environments. This enables sLMs to learn how to act for solving tasks.
2.2 Language agents and agentic reasoning
An agent can be broadly defined as an entity that autonomously pursues goals by observing the world and acting upon it. Powered by LLMs, early works like ReAct [20, 33] introduced the concept of language agents-wiie which observe the world, think natural language, and act to complete the diverse range of tasks interactively. Since most LLMs not natively trained for such interaction, prior works have relied on carefully designed prompts (e.g.. few-shot examples) for stronger LLMss, and fine-tuned weaker LLMs trajectories from stronger ones [20, 33-42). Building on foundations, recent works have pushed language agents toward more advanced agentic capabilities.
Farly works focus teaching LLMs to use tools that enable interaction with external environ- ments [43-47]. Furthermore, agentic retricval systems have emerged to support multi-hop reasoning over real-world knowledge [48-50], while tool-augmented reasoning leverages external capabilities like code exccution to tackle challenging math problems [51-55]. Other approaches promote the notion of agentic reasoning, enhancing the decision-makin and planning capabilities of LLMs for solving complex tasks with tools through prompting or reinforcement learning [56-58].
Unlike prior work, which primarily focused on fine-tuning LLMs (2 7B) on trajectories from stronger close-sourced LLMs (e.g., GPT-4 [12] in FireAct [34]), our work aims to distill the agentic capabilities of L.MMs into much smaller models (sLMs, & 3B), enabling them to operate as capable agents. We address key challenges such as improving the quality of teacher trajectories and optimizing student behavior at test time, building on improved agent framework [21]. We show its effectiveness
3

---

across a range of small models (e.g., 0.5B-3B) and tasks requiring strong knowledge and reasoning capabilites-an an under-explored yet important setting for practical, small language agents.
3 Preliminary
Knowledge Distillation. Knowledge distillation [59] transfers the capabilities of a large teacher model pr to smaller student model ps. Modern language models follow the auto-regressive transformer architecture [60], where a token-level policy predicts the next token given previous tokens. Given source and target sequences (æz, y), distillation optimizes the following objective:
by
min E (m. Dusin
D(pr(- yen. x)
|| ps(- Ven, x; 0)),
Ly n=1
where D is a divergence metric (e.g.. Kullback- or Jensen-Shannon Shannon divergence), and Ly denotes the length of the target sequence y.
Reasoning distillation. In reasoning tasks, the target sequence y can be rationale that solves the problem step-by-step. Since collecting proaches [16, 24, 25, 27] use chain-of-thought (CoT) prompting [23] to generate rationales with large teacher models and train the student to imitate them:
reasoning is expensive, recent ap-
Ly
min - Eng D ( ICoT)
ps yen 0),
(2)
x=1
where IcT denotes a CoT-style prompt such as "Let's think step by step." [23].
Agent Distillation
While reasoning distillation is effective and has become a standard post-training technique [11, 1], it does not equip models with the ability to interact with external environments through actions. Recent work [20, 21] shows that large models can generate actions grounded in intermediate reasoning. observe feedback from the environment, and adapt accordingly.
We refer to such interactive sequences as agent trajectories, consisting of repeated cycles of thought (r), action (a), and observation (o). Given an input æ, the teacher model generates trajectory:
((r1. d1, 01), (rL,, aL, aL,; OL.)) pr(- x, Lagent),
where aggnt is an instruction prompt for the agent (e.g., "To solve the task, you must plan forward to proceed in a series of steps, cycle of Thought: Code:, and Observation: sequences" [21, 61]). Each observation o comes from the environment in response to action a. not generated by the model.
Following prior works [34, 52], we fine-tune the student model on generated trajectories, excluding observations from the loss:
min Em D Domn. Poor Tagon)
(rg |x, TTT
(4)
where Tex ((ri, a, oi)..... (rr-1, a, 1))
This distillation enables student models to function as interactive agents. For instance, model distilled from CodeAct [21] can reason about which code snippet to generate, generate actions as codes (e.g., API calls, loops), and respond to execution feedback. If the interpreter returns an error, the model can revise the code accordingly; if the output is valid but insufficient (e.g., suboptimal search results), can rephrase the query and continue the task adaptively.
Despite its promise, agent distillation presents two key challenges, particularly when applied to small language models (sLMs). First, agentic behavior often lies distr--distriuution relative the pre- training and instruction-tume tuning distribution of both teacher and student models. As a result, distilling such behavior may degrade performance on domains where the student already well-optimized for CoT--style reasoning. Second, although sLMSs are pretrained on large code corpora [62], they may struggle to produce functional code during inference. Typical failure cases include misformatted code outputs or incorrect usage of library functions, which hinder the ability of agents to interact.

---

Think step
Identify the
Selected path
Code: (220y4) / (x20x0))
Interpeste Ente
by step
problem
direction
Col' Prompt
Teacher
Mistt-houught prefix
11.5
You
Trought: To solve this problem....
1. The slope of
Question:
(92- xa3
le solre
Luse can found
Fad eoooputte
Code;, usiaz
:hought:
tick, prosa
staps of
Teacher A
Distill
xa, ya
x3,
Code:
of pones 11)
Prompt
Codez stope (2 (-1) (30
1.5 Consessent Observatio
uols....
Coder xh, X0 8, 10 a x2-a1
Agent
StudentA
Prompt
(a) First: thought Prefix
(b) Self consistentA ction Generation
We prompt teacher with CoT prompt to induce step-hy-step reasoning. The first reasoning step used a prefix to generate agentic trajectory. which is then distilled to student agent teach CoT-style reasoning initialization. (b) Self-consistent Action Generation: The agent generates multiple candidate actions and selects the one with consistent outcomes. Thoughts are omitted for brevity.
Figure 3: (a) First-thought Prefix:
First-thought prefix. We observe that instruction-tuned L.MSs (e.g., (wwe2.55-32B-Instrul [1], when employed as agents, demonstrate reduced performance on challenging problems from MATH500 benchmarks compared to their performance with CoT prompting [23] (see Appendix D.1 for experimental results). This degradation can further propagate during distillation, negatively impacting student models where they have also been instruction-tuned on CoT-style data.
We hypothesize that instruction-tuned models, which have already been trained to produce CoT reasoning to solve the task, can exhibit distributional drift when prompted with agent instructions (e.g.. Prompt D.2 in Appendix). Although these models are capable structured reasoning, the additional instruction generate reason-act trajectories may override or conflict with their original reasoning patterns. As a result, the model may deviate from the correct reasoning path it would otherwise follow under CoT prompting. Since prior studies have shown that the initial reasoning step critically determines the final conclusion of LLMs [63-65], ensuring that the model begins reasoning in an appropriate direction during its first action generation becomes essential for maintaining accurate reasoning.
To this end, propose the first-thought prefix (FTP). Motivated by the prefix-attack in LLM jail- breaking works [66-68], this method integrates the initial reasoning step from a CoT prompting as prefix the agent's first thought in Figure 3(a). Formally, we modify the trajectory sampling described in Equation 3 as follows:
(5) where yı is the first-step of CoT reasoning and r' denotes the completed first thought of the agent following the prefixed first-step yı- Note that this method is only used to generate trajectories from the teacher agent; the student agent does not explicitly require first-thought prefix during inference.
p- IC.T). T= (f., aL. ~ pr U1. Iagent);
Self-consistent action generation. We observe that small distilled agents often produce invalid actions, particularly in the context of CodeAct [21], where invalid actions refer to code that either fails to execute or throws errors. To improve robustness in action generation, we introduce self self-consistent action generation (SAG). Instead of using greedy decoding, we sample multiple N thought-action sequences for each step through nucleus sampling [69] with a high temperature to encourage diversity. We then filter out any sequences that result parsing or execution errors using a lightweight code interpreter. When all generated actions fail, retain one randomly selected failed action and feed its error message back as an observation, allowing the model to self-correct in subsequent steps. To further ensure correctness, the action whose output is most consistent across samples. For example, in Figure 3(b), the agent generates four candidate sequences. One result in an interpreter error is filtered out. Among the remaining three, two produce the same output, so we select one of these two consistent actions as a final action.
perform majority voting over the resulting observations [70], selecting
Experimental setup
We evaluate our proposed Agent Distillation across benchmarks to test whether small language models (sLMs) can acquire agentic abilities from large language model (LLM) agent teacher.
5

---

Table 1: Task categorization with domain and sampled test data size used.
Task Type
Domain
Dataset Name
Description
Test Data Size
In-domain
HotPotQA [2]
2-hop question-answering
Factual
Out-of-domain
Bamboogle [3]
2-hop question-answerie
125
Reasoning
Out-of-domain
MuSiQue [4]
3-hop -aasstionnnnwering
500
Out-of-domain
2WikiMultHHopQA [5]
2-hop question-answente
500
In-domain
MATH [6]
College-level math
500
Math
Out-of-domain
GSM-Hard [7]
Large number arithmetics
500
Reasoning
Out-of-domain
AIME [8]
Olympiad-leve problems
90
nut-of-domain
OlymMath [9]
Olympiad-lew problems
200
Tasks and datasets. We evaluate two categories of reasoning tasks: factual and mathematical. For each, we assess both in-domain and out-of-domain generalization. We use 1,000 HotPotQA [2] and 2,000 MATH [6] examples for training. For test benchmarks, summarize them in Table To reduce evaluation cost, we limit each test set to 500 examples, following Wang et al. [71]. As metric, we use exact match for math and llm-as-a-judge [72] using gpt-4o-mini for factual reasoning.
Models. The teacher model Qwen2. 1--22-Innstrutt 323 parameter instruction-tune model. For student models, we use the Quen2. 3--Intruue series with four sizes: 0.5B, 1.5B, 3B, and 7B parameters. All student models are instruction-tune prior distillation [1].
We compare two main distillation paradigms: (1) CoT distillation [16], which transfers static reasoning traces generated using Chain-of- -fain-of-Thouuht prompting, and (2) proposed Agent Distillation, which transfers interactive reason-acoe obbeeee trajectories. For CoT distillation, we add the baseline that uses aarreval-augmennted generation [73] in both distillation and inference for a fair comparison with external knowledge [19, 28, 29]. For ours, we adopt the formulation from CodeAct [21, 61], where each step consists of Thought, Action (e.g., Python code), and Observation. Additionally, we incorporate two proposed methods - distillation using trajectories through first-thought prefix FTP and self-consistent action generation SAG.
Baselines.
Training& & inference details. For reproducibility of experiments, we use Wikipedia 2018 as knowledge base for both agents and RAG instead of search engine. We use e5-base-v2 [74] as both document and query embeddings as Jin et al. [49]. For both CoT and agent, we sample one trajectory question from the teacher model and filter out wrong trajectories, resulting in approximately 2,000 trajectories for distillation.
We fine-tune student models using parameter- efficcient tuning with LoRA (rank 64) on all linear layers [75]. All models are fine-tuned for 2 epochs using a batch size of 8 and learning rate of 2 20 All experiments are conducted using four NVIDIA A100 80GB GPUs.
For inference, use greedy decoding. For all agents, experiments, we set N with temperature to 0.4. More details are in Appendix C.
steps to 5. For SAG in main
Results
Overall results. In Table 2, we find that agent distillation consistently improves performance across all model sizes. Before distillation, most sizes of models (except 7B) fail to produce effective agentic outputs via prompting alone, often generating incorrect or unparscable code action. In contrast, our distilled agents outperform CoT-distilled counterparts, particularly tasks across both factual and mathematical domains. These results highlight the effectiveness of agent distillation in improving generalization of sLMs. Notably, the gains are further amplified by our two methods-Firss First-thought Prefix (FTP) and Self-consistent Action Generation (SAG).
out-of-domain
Our findings also demonstrate that agent distillation enables small models to match or exceed the performance of CoT-distilled models that are 2-4x larger, offering a promising path toward efficient and capable language agents. Specifically, the 0.5B agent matches the performance of a 1.5 CoT-distilled model, the 1.5B agent reaches its 3B counterpart, the 3B agent surpasses the 7B CoT model, and the 7B agent outperforms the 32B CoT model.

---

Table 2: Main results. Distilled agents show the strong performance most of tasks, especially out-of- domain tasks, compared to baselines. FTP First-Thought Prefix, SAG Self-consistent Action Generation.
Highlighting best among same-sized models. Avg. denotes the average score across tasks.
In-domain
Out-of-domain
HotPot
MATH
MuSi- Que
Bamb- oogle
2Wiki
GSM- Hard
Olym- MATH
Params
Method
500
AIME
Avg.
Teacher: Qwen-2.5 1ieet-..-Instruct
CoT
Prompting
36.8
79.2
12.2
60.8
33.4
74.6
13.3
6.0
39.54
32B
Agent
Prompting
56.4
69.2
25.2
58.4
49.8
76.4
21.1
11.5
46.00
Student:
Qwen-2.5-Inttruct
Prompting
29.2
71.8
5.8
43.2
29.2
66.6
12.2
33.19
CoT
Distill
31.0
72.6
9.0
44.8
26.8
67.6
10.0
6.5
33.54
Distill + RAG
42.8
68.0
6.6
40.0
27.6
60.6
6.7
5.0
32,16
7B
Prompting
46.8
56.0
16.8
41.6
45.6
62.2
13.3
10.0
36.54
Distill
51.2
62.2
19.6
$2.0
45.2
72.0
11.1
5.5
39.85
Agent
ETP
55.0
66.6
17.6
56.0
44.6
70.8
14.4
13.0
42.26
53,2
64.0
20.6
50.4
73.4
15.6
41.86
PTP3AC
54.4
67.8
19.4
55.2
45.2
72.4
13.6
11.5
42.68
Prompting
38.6
62.8
33.6
21.6
60.2
6.7
29.27
CoT
Distill
26.8
61.8
6.4
34.4
25.0
56.8
5.6
5.0
27.72
Distill + RAG
40.6
59.6
4.6
32.0
28.2
53.2
5.6
4,5
28.53
Prompting
38.6
30.5
8.8
29.6
28.8
25.8
4.4
3.0
21.20
Distill (Ours)
48.4
54.0
13.0
37.6
37.4
64.2
6.7
7.5
33.60
Agent
ETP
47.6
54.4
13.0
43.2
41.4
63.0
7.8
34,49
48.6
$7.4
13.0
36.0
37.4
65.6
10.0
33.50
PTPEAC
49.4
60.2
15.8
38.4
41.0
65.4
13.6
7.0
36.60
Prompting
17.8
47.6
21.6
19.0
49.0
1.1
20.33
CoT
Distill
23.8
46.4
2.0
21.6
18.4
51.0
5.6
1.5
21.28
Distill RAG
37.6
48.6
4.2
26.4
27.0
48.6
2.2
2.5
24.64
1.5B
Prompting
8.6
22.2
1.6
10.4
10.6
9.0
1.1
0.0
7.94
Distill (Ours)
43.0
46.8
9.0
27.2
35.6
54.8
1.1
7.0
28.06
Agent
FTP
43.6
46.4
30.4
32.6
60.6
7.8
29.11
43.8
49.8
11.6
31.2
36.6
$8.0
7.8
3.5
30.29
ETPSAC
45.6
50.6
9.2
33.6
33.6
60.6
6.7
4.5
30,55
Prompting
9.2
28.4
0.2
7.2
12.8
25.6
1.1
11.06
CoT
Distill
13.2
28.6
1.4
10.4
23.8
28.6
1.1
2.0
13.64
Distill RAG
29.2
28.0
1.6
13.6
25.4
27.4
0.0
2.0
15.90
0.5B
Prompting
2.4
3.0
0.0
0.8
2.8
5.4
0.0
0.0
1.80
Distill (Ours)
34.6
30.4
7.0
17.6
28.8
31.2
3.3
19.24
Agent
32.4
28.8
24.0
36.4
34.0
33.8
13.6
33.0
33.0
4.4
20.01
ETPSAQ
33.4
34.4
5.6
24.0
31.2
40.8
3.3
2.5
21.90
Table Comparison of performance across general and cade-specific models. 32B/1.5B denote general models 5B-Coder denote specifcc models. For all models, we apply SAG with 8.
HotPot
MATH
MuSi-
Bamb- oogle
2Wiki
GSM-
Olym- MATH
Teacher
Student
AIME
Avg.
QA
Que
QA
Hard
32B
1.B
45.6
50.6
9.2
33.6
33.6
60.6
6.7
4.5
30.55
32B-Coder
1.5B
42.6
51.4
10.0
36.8
36.8
60.0
6.7
3.0
30.91
32B
1.5B-Coder
37.8
52.6
8.2
30.4
38.0
59.8
3.3
6.0
29.52
32B-Coder
1.BB-Coder
41.4
49.2
30.4
37.4
63.6
4.4
5.5
30.17
Factual reasoning results. We find that retrieval improves the performance of CoT-distilled models on factual reasoning benchmarks. However, due to its static nature, it can degrade performance on tasks requiring dynamic or adaptive information use, such as mathematical reasoning. In contrast, our distilled agents outperform even RAG-enhanced CoT models. This is because agent distillation equips the model actively retrieve and integrate knowledge during reasoning, rather than relying solely pre-fetched documents that may be insufficient misaligned with the task.
Math reasoning results. mathematical reasoning tasks, our distilled agents demonstrate strong overall performance. The 1.5B, 3B, and 7B models show improvements on the AIME and Olym-

---

Table 4: Comparison of performance Llama-3. 2-1B-Instruct [11] and Phi-4-mini--nsttie [76]
models. Teacher model Qwen- 5--3---I---ntree Performance trends are consistent results Table
HotPot
MATH
MuSi-
Bamb- oogle
2Wiki
GSM-
Olym- MATH
Student Model
Method
AIME
Avg.
QA
500
Que
QA
Hard
CoT
Prompting
13.2
28.8
1.2
14.4
8.0
19.0
1.1
2,5
11.53
Llama-3,2-18
Distill
18.2
25.6
2.6
25.6
19.0
13.8
1.1
2.0
13.23
-Instruct
36.0
34.6
2.6
11.2
26.4
40.4
1.1
19.54
Agent
37.6
32.8
3.6
24.0
30.8
45.0
1.1
1.5
22.93
PTPSAG
40.6
40.0
3.2
23.2
30.0
47.8
1.1
3.0
23.97
CoT
Distill
Prompting
24.2
24.4
53.8
5.8
6.0
38.4
24.8
24.2
54.8
49.6
6.7
5.6
4.5
25.04
27.41
Phi-4-mini -nnstrcct (3.8B)
Distill
48.2
8.8
27.2
33.6
69.4
5.6
31.52
Agent
FTF
45.2
60.0
7.2
34.4
39.2
71.2
10.0
7.5
34,58
FLFBAC
47.0
9.6
32.0
41.0
73.0
11.1
35.79
CoT
Agent
Agent FTP
Performance by Problem Category
Performance by Problem Level
Accuracy (%)
50
50
25
Algehex
Couningg Probubility
Goometty Intermediate
Namber
Proilgshas Procalenius
1
3
5
Algehra
Thoor
Level
Category
Figure 4: Performance comparison on the MATH subcategories and levels between CoT and Agent distillation of 3B models. Left: Accuracy by problem category. Right: Accuracy by problem difficulty level. The results highlight that FTF improves the performance of small agents in harder problems.
MATH benchmarks, benefiting from code tool use for complex calculations acquired through distilla- tion. On GSM-hard, agent distillation improves robustness in reasoning over rare number combina- tions, such as 6-digits arithmetic. While performance on MATH500 lags behind CoT-distilled models in 3B and 7B models, we attribute this to the Qwen2.5 series being heavily Instruction-tunee on college-level math, which may align better with COT. Furthermore, we conjecture that larger models (3B and 7B) possess stronger internal computation skills, making tool use less beneficial on bench- marks like MATH500, while smaller models benefit more from external code execution. Nonetheless, agent distillation remains effective for larger models on harder math problems (e.g.. GSM- Hard, OlymMATH), where the agentic method yields consistent gains. Overall, agent distillation delivers substantial gains across wide range of math tasks. We provide a detailed breakdown in Section 7.
7 Analysis
Code-specific teacher yields better caddentt--marginalyy. We primarily study general instruction- tuned models for both the teacher and student agents, as shown Table Given that CodeAct [21] requires generating code to perform actions, a natural question arises: Can we obtain better agents by using code-specific models for the teacher or student in the agent distillation process?
To explore this, we conduct the same set of experiments using 3-co22s-o-orr332---111 the teacher and Saenn2.5-Coder--- 1et---1..BInnete Instrutt the student [77]. The results, presented in Table 3, suggest that the use of a code-specific student model does not significantly impact performance. Instead, the choice of a code-specific model as the teacher appears be more influential in generating effective trajectories for distillation. Nevertheless, the overall improvements are marginal average, indicating that code-specific post-training has limited impact, which suggests the code knowledge is not critical bottleneck of the student.
Agent distillation applies across different model families. We further validate whether the improvements from our method generalize across different language model families. In Table 4, we conduct experiments with two additional student models, .lama-3.2211 2-1B-IInttruct [11] and Phi-4-mint instruct [76]. Results show that both models benefit from agent distillation com-

---

Col
Agent
CoT
Agent
Math
Hotpo:QA
15
100
OuT Moan: 179.3
100
CuT Mcan: 591.3
Agent Mean: 267.1
gent Mean:
10
Frequency
Auguracy
0
0
HolpotQA
MuSiQue
MuSiQue
AIME
80
15
OUT Moan: 231.0
CoT Menn: 1022.5
40
Agent Mean: 190.2
Agent Mean:
10
2
500
1000
2000
4000
Conll-Conistencyy Sample Size)
Token Count
Figure 5: Comparison of SAG in agents and self-consistency [70] in CoT for 3B models: self-consistency in CoT is help- ful in math tasks but not in factual tasks.
Figure 6: Generated token counts comparisons 3B models. For factual reasoning tasks (HotpotQA, MuSiQue), agent gener- ates more tokens than CoT. In contrast, for math reasoning tasks (MATH, AIME), CoT generates slightly more tokens than agent.
pared to CoT distillation, Moreover, both FTP and SAG yield consistent improvements across the two models, demonstrating that our proposed method is broadly applicable to different model families.
First-thought prefix improves the agents on more complex reasoning problems. In Table 2, we observe that agent distillation does not improve performance MATH500 compared to CoT distilla- tion, particularly for the 3B model. To investigate further, we break down MATH500 performance by both problem category and difficulty level.
Interestingly, naive distillation degrades the performance of distilled 3B agent on most of levels. However, when using teacher trajectories with first-thought prefix, distilled 3B agent shows improved performance on level and 5 problems- These results suggest that trajectories from FTP help student agents become more robust on complex reasoning tasks, a trend also observed in the challenging AIME benchmark in Table 2.
especially significant gains at level 5.
However, a remaining concern is the performance drop in certain categories most notably, a decline in precalculus, Our analysis suggests that this degradation is primarily due to the nature of certain problem types that require an analytic approach rather than straightforward calculations (e.g., applying properties of trigonometric functions). Such problems are harder to solve using code tools. We explore this issue in detail in Appendix D)
Self-consistency improves CoT, but the agent with SAG still performs better. Self-consistent action generation (SAG) enhances small agents by filtering out invalid code actions and retaining only those that are consistent with observations. Similarly, self-consistency [70] can be applied at test time in Thain-of-Thought (CoT) reasoning to improve performance without relying on an external verifier.
natural question is whether CoT with self-consistency, using the same computational budget, can outperform an agent with sAG. To investigate this, we conduct experiments using seff-consistency [70] on CoT-distilled small language models (sLMs), applying majority voting over multiple samples.
As shown in Figure 5, in the MATH senchmark-wher CoT already surpasses the agent with SAG- self-consiseency further improves the performance of the CoT-distilled model. However, in the more challenging AIME benchmark, the small agent with SAG still outperforms the CoT-distilled model un- der the same generation budget. Moreover, in factual reasoning tasks such HotpotQA and MuSiQue, self-consistency yields only marginal gains, suggesting limited effectiveness in these settings.
How many tokens should agents generate? A natural question is whether distilled agent should generate significantly more tokens than a CoT-distilled model, potentially affecting the efficiency and practicality of small models. To investigate this, we analyze token counts on two factual and two math reasoning tasks using 3B distilled models.
As shown in Figure 6, there is no significant difference in total token generation between the two approaches across both domains. factual reasoning, the agent tends to generate more tokens due to making multiple retrieval calls across several steps to gather accurate information. In contrast, in math reasoning, the agent generates fewer tokens than CoT models by delegating repetitive calculations to code execution, often leveraging logical structures like for-loops.

---

0.5B Parse
1.5B Parse
Parse
Without FTP
FTP
0.5B Exec.
1.5B Exec.
3B Exec.
Nyange, Retrigval Calls
2WSESQA
3.5
Damboogle
MATH
GSM-Hard
AIME
3.0
Error Rate (%)
2.5
50
20
5
HerporQA
MuSiQue
SAG w/ SAG
w/o SAG
SAG
w/o SAG w/ SAG
0.53
1.58
Model Size (D)
1.5B
Figure 7: Impact of Self-consistent Action Generation (SAG) on code generation errors across models and math datasets. BaG consistently reduces code parse (dark) and code execution (light) errors, especially for smaller models (0.5B) and the AIME dataset.
Figure Average retrieval tool calls across three model sizes and datasets. Harder tasks and larger sizes make agents use more retrieval calls.
SAG significantly reduces invalid code actions. In Figure 7, we show the effect of coff-sonsistent action generation (SAG). SAG reduces the generation of codes with both parsing and code execution errors. This result indicates that the small distilled agent is capable the likelihood of generating valid code tends to decrease with smaller model sizes. SAG mitigates this issue by sampling multiple actions per turn, increasing the likelihood of generating a valid one. Nevertheless, execution errors may still occur, in which case the agent uses the error message as feedback to revise its code in the next turn.
generating valid code but
Larger agents make more retrieval calls, FTP reduces them We analyze how frequently agents use the retrieval tool across different model sizes and factual reasoning benchmarks. As shown in Figure 8, larger models tend make more retrieval calls than smaller ones, likely because they are better distilled from teacher trajectories and more effective at formulating queries and deciding when to retrieve information. In contrast, smaller models may underuse retrieval due to weaker judgment or limited capacity. For instance, they often over-rely on an initially retrieved document, even when it lacks the necessary information, rather than attempting a new retrieval.
Interestingly, we find that the first-token prefix (FTP) leads agents to make fewer retrieval calls. As shown in Table 2, FTP improves performance in Bamboogle, but results are mixed in HotpotQA and MuSiQue, possibly due to reduced retrieval. One explanation is that FTP encourages generating factual statements in thought process, which can lead agents-especiall internal knowledge instead of retrieving them, increasing the risk of hallucination. These findings suggest that the composition of teacher trajectories plays a crucial role in helping student models learn effective tool use, especially for solving complex tasks. We include more analysis in Appendix D.
smaller ones-to utilize their
8 Conclusion
We proposed Agent Distillation, framework for transferring agentic behavior and tool use from LLMs to small language models (sLMs). By introducing first-thought prefix and self-consistent action generation, we improve both the quality of teacher trajectories and student robustness at test time. Our experiments show that distilled small agents can match or outperform next-tier larger models trained via CoT distillation, especially on oft-of-domain tasks. These results highlight agent distillation as a practical path for building capable, tool-using small models for real-world problems.
Limitations & Future Works. While our method shows strong overall performance, it also highlights several open challenges. The first-thought prefix (FTP) improves agent distillation on average, underscoring the importance of high-quality teacher trajectory generation for effective distillation. However, FTP can sometimes degrade performance, especially when the model generates facts during reasoning instead of leveraging tools (Figure 8). This highlights the need for improved agentic trajectory generation strategies that align with the behavior and limitations of small models.
The success of self-consisted action generation (SAG) (Figure 7) suggests the potential of test-time compute scaling and opens up opportunities for incorporating process- reward models [78, 79].
Finally, while agent distillation enhances the sLMs through agentic behavior, it does not directly improve their core reasoning abilities. Reinforcement learning tool-augmented environments [48, 54, 80] could further refine these models post-distillation across diverse domains.
10

---

Acknowledgment
This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (ITP) grant funded by the Korea government(MSIT) (RS-2019-11190077)) Artificial
Intelligence Graduate School Program (KAIST)), the Institute of Information & Communications Technology Planning & Evaluation (ITP)) with grant funded by the Ministry of Science and ICT (MSIT) of the Republic of Korea in connection with the Global AT Frontier Lab International Collab- orative Research. (No. RSS2024-004694822 & RS-2024-00509277,, National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS 00.2022,,,,,,,55), and the KRAFTON AI Research Center.
References
[1] An Yang. Baosong Yang. Beichen Zhang. Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv, 2412.15115, 2024. URL hattps:///ol org/10. 48550/ arriv.. 2412. 15115.
[2] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun' ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2380, Brussels. Belgium. October-Novemteee 2018. Association for Computational Linguistics, doi: 18.18//3/v 1/DI8 1 URL https: / acl anthology 2eg/D18-12555
[3]
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Measur- ing and narrowing the compositionality gap in language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore. December 6-10, 2023, pages 5687-5711. Association for Computational Linguistics, 2023. URL https: /di.. org/10 10. 18653/v1/2022 findings - emnlp 378. e78.
[4] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Trans. Assoc. Comput. Linguistics, 10:539 554, 2022. URL https: org/ wttp/t1c11_ea00000 a_0475..
[5] Xanh Ho, Anh-K Khha Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi-hop QA dataset for comprehensive evaluation reasoning steps. In Donia Scott, Núria Bel, and Chengqing Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 6609-6623 International Committee Computational Linguistics, 2020. URL https: /ttp:::////oooo///...../ wtt.://oo......//////////// 18653/v1/2020 coling-main. 580.
[6] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving Joaquin Vanschoren and Sai-Kit Yeung, editors, Pro-
with the MATH dataset. ceedings of the Neural Information Processing Systems Track on Datasets and Bench- marks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. www.eekdis-pooeeeiingg.... wwwriis.cccpapeeeeeeeeeeeeeeeeee wttp:ccpppppppppeee cccppaper/222//hes/ URL https: aBBabbeecbbb77aa22ccbb0.7788888 -wwt-rbtsttttoooeee round2. .ttml.
[7] Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL:: program-aided language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 10764-10799. PMLR, 2023. URL
mlr gress/v202, hatp///2222////////
[8] AI-MO. Aime. https://hugginggee http://nnngacencccttttttteeeeeee wtt..//ttttttttteatatttt... waw.imo-iiaaaaiiiiiiiiiaa. 2024.
11

---

[9] Haoxiang Sun, Yingqian Min, Zhipeng Chen, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang. Lei Fang, and Ji-Rong Wen. Challenging the boundaries of reasoning: An olympiad-level math benchmark for large language models. arXiv, 2503.21380, 2025. URL https:/p:: htttts://doi.oog/10.. 48550/az /arivi///vv.v/.. 2503. 21380.
[10] DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang. Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue. Bingxuan Wang. Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu. Haocheng Wang, Honghui Ding. Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo. Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang. Mingming Li, Ning Tian, Panpan Huang, Peng Zhang. Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang. Ruizhe Pan, Runji Wang, R. J. Chen, R. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang. Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao. Wengin Yu, Wentao Zhang. W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang. Xiaokang Chen, Xiaotao Nie, Xin Cheng. Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song. Xinyi Zhou, Xianzu Wang. Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang. Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang. Yukun Zha, Yuting Yan, Z. Z. Ren, Zchui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu. Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement /ttp///axvvvvv.///////e./...av./ org/ /5011 12948.
learning. 2025. URL
[11] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, Anirudh Goyal. Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis. Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino. Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve. Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jeimer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang. Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The llaa 3 herd of models. arXiv, 2407.21783, 2024. URL https: / //doi.org/10 48550/ 2407 21783.
[12] OpenAL. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023. URL https: aidoi.org/1e wtt.:///g//...////////////////// 48550/arXiv. 2303 13003.000 08774.
[13) Zechun Liu, Changsheng Zhao, Forrest N. landola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, Liangzhen Lai, and
12

---

Vikas Chandra. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases. In Forty-first International Conference on Machine Learning. ICML 2024, Viemna, Austria, July 21-27, 2024. OpenReview.nnt, 2024. URL. htttps://opeerevtaee wttpeee..eeeeeeeee net/forum? id=EIGbXbxcUQ.
[14) Jinheng Wang, Hansong Zhou, Ting Song, Shaoguang Mao, Shuming Ma, Hongyu Wang, Yan Xia, and Furu Wei. 1-bit AI infra: Part 1.1, fast and lossless bitnet b1.58 inference cpus. arXiv, 1610. 2024. URL https://ddd org/10 1000 48550/ arr.vv/rrvvvvv.. 16144.
[15] Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, Xiwen Zhang, Nicholas D. Lane, and Mengwei Xu. Small language models: Survey, measurements, and insights, arXiv, 2409.15790, 2024. URL https::/dol httt::::/ddi.oog////// 10. 48550/arXiv
2409 15790.
[16] Namgyu Ho, Laura Schmid, and Se- Young Yun. Large language models are reasoning teachers. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 6l Annual Meeting of the Association for Computational Linguistics /Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 14852 14882. Association for Computational Linguistics, 2023. URL https:tp//// /tp///o....// orggro///////. 18653/v1/2023 act-::....ccloooooo acl-long. 830.
[17] Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of GPT-4. arXiv, 2306.02707, 2023. URL https://doi. orgooo///// 10. 48550/arXiv 2306. 02707.
[18] Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle learn long-tail knowledge. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 1 15696-15707. PMLR, 2023. URL html.
mlr
[19] Minki Kang, Seanie Lee, Jinheon Baek, Kenji Kawaguchi, and Sung Ju Hwang. Knowledge- augmented reasoning distillation for small language models in snowledge-intensivee tasks. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http: http://papers: /twppappppepaaaaoee nips cc heep/ppp__fiiiie files/ www.fffaateee///eeeeeaeaaaaaeeee httpp/pppppp///////ea/ 7
Abbrrratttttttt... Conference, html.
[20] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: httthtts///ppeerrr/e/ //tpp//rv/////e/e///////e wttiiiiwwww.eeeeeeee net/forumiid-wer. -
[21] Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better LLM agents. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: / opw.revieeereeeeeeeeeeee wtte://foommmmidjJBBooooo..
[22] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-22213, 2022.
[23] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thougle prompting elicits reasoning in large language models, In Advances in Neural Information Processing Systems, 2022.
[24] Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. Specializing smaller language models towards multi-step reasoning. arXiv preprint arXiv:2301, 12726, 2023. URL https:
/ //doi.oor/// http://rr/r....5////111...
[25] Cheng- Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen- Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Findings of the Association
13

---

for Computational Linguistics: ACL 2023, pages 8003-8017. Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v12022..nnnil Sindings-acl.5000 URL httts://aclaatthen
org/ 2023/2023.22 fiidiiggs-acll acl.inn.-acl.0...
[26] Hojae Lee, Junho Kim, and SangKeun Lee. Mentor-kd: Making small language models better multi-step reasoners. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami. FL, USA, November 12-16, 2024, pages 17643-17653 17658. Association for Computational Linguistics, 2024. URL tttps// http:///ccaaathooooooooo www.gg.....gggge org/2024 emnlp main.
977.
[27] Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models to reason. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1773-1781, Toronto, Canada, July 2023. Association for Computational Linguistics.
[28] Xiang Li, Shizhu He, Fangyu Lei, Jun Yang Jun Yang. Tianhuang Su, Kang Liu, and Jun Zhao Teaching small language models to reason for knowledge-intensive multi-hop question answer- ing. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 7804-7816. Association for Computational Linguistics, August 2024. URL https: http://aaaaaathhho
org/2024. 2024. f findings-ach acl. 464//
[29] Shengmin Piao and Sanghyun Park. TinyThinker: Distilling reasoning through coarse-to- tineine knowledge internalization with self-reflection. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pupers), pages 6069-6087. Association for Computational Linguistics, April 2025. URL https: / acc/acllllllll hatp//oooanttoooyyyyoooo org/2025 naacl-long 309/.
[30] Minki Kang, Jongwon Jeong, and Jaewoong Cho. T1: Tool-integrated self- for test-time compute scaling in small language models, 2025. URL https://arxxv.orgg 2504.04718.
[31] Xuekai Zhu, Biqing Qi, Kaiyan Zhang, Xinwei Long, Zhouhan Lin, and Bowen Zhou. Pad: Program-aided distillation can teach small models reasoning better than chain-of-thought fine- tuning. In Kevin Duh, Helena Gómez-Adorn, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Volume I: LLnng Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pages 2571 1-797.. Association for Computational Linguisties, 2024.
[32] Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. Distilling mathematical reasoning capabilities into small language models. Neural Networks, 179:106594, 2024. URL https: neunet. seunet.2024...... 106594.
/ //do/.org//0.
[33] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Ad- vances in Neural Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http://papers: .ips.. wac/paper_ffee caarrsffls//ae www.eeeaaeee/eeeeeeeeaaaaaaeeeee pttppp/ppp////////eapp -twp:/110000-00--raaaeeaee Conferrnnneeeeeeeeeeeeeeeee html.
[34] Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning. arXiv, 2310.05915, 2023. URL https:
/ oogoi.org/11 48550/ arXiv. 2310. 05915.
[35] Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. Agent-flan, Designing data and methods of effective agent tuning for large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics, ACI. 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 9354-9366 Association for Computational Linguistics, 2024. URI. https:/ / /oi.. org/10.11 18653/663//1 /0t//v122222/2 findings-acllli 557.
14

---

[36] Arindam Mitra, Luciano Del Corro, Guoqing Zheng, Shweti Mahajan, Dany Rouhana, Andrés Codas, Yadong Lu, Weige Chen, Olga Vrousgos, Corby Rosset, Fillipe Silva, Hamed Khanpour, Yash Lara, and Ahmed Awadallah. Agentinstruct: Toward generative teaching with agentic flows. arXiv, 2007 .0550, 2024. URL.https://da org/10. 48550/ arriiv 2407 03502.
[37] Shuofei Qiao, Ningyu Zhang, Runnan Fang, Yujie Luo, Wangchunshu Zhou, Yuchen Elcanor Jiang, Chengfei Lv, and Huajun Chen. Autoact: Automatic agent learning from scratch for QA via self-planning. In Lnn-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties ( (Volume 1: Long Papers), ACL. 2024, Bangkok, Thailand, August 11-16, 2024, pages 3003 3021. Association for Computational Linguistics, 2024. URL https:://det org/ 10. 18653/ /0/553//1/20 v2/2024 acl -ong. 165.
[38] Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Raghavi Chandu, Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin. Agent lumos: Unified and modular training for open-source lan- guage agents. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 12380-12403, Association for Com- putational Linguistics, 2024. URL https: //doi. org/ 10. 18653/ 201533/11/0224 acl sel-longg. 670.
[39] Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for Ilms. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeling, August 11-16, 2024, pages 3053-3077. Association for Computational Linguistics, 2024. doi: 10.18653/V https: //tos: org/10. 11w53/v1/ 1at.///11/222222 findings- acl 181.
URL
[40] Jianguo Zhang. Tian Lan, Rithesh Murthy, Zhiwei Liu, Weiran Yao, Juntao Tan, Thai Hoang, Liangwei Yang, Yihao Feng. Zuxin Liu, Tulika Manoj Awalgaonkar, Juan Carlos Niebles, Silvio Savarese, Shelby Heinecke, Huan Wang, and Caiming Xiong. Agentohana: Design unified data and training pipeline for effective agent learning. arXiv, 2402.15506. 15506, 2024. URL https://doi. org/10. 48550/arXiv 2402. 15506.
[41] Qinhao Zhou, Zihan Zhang, Xiang Xiang, Ke Wang, Yuchuan Wu, and Yongbin Li. Enhanc- ing the general agent capabilities of low-paramter LLMs through tuning and multi-branch reasoning. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Findings of the As- sociation for Computational Linguistics: NAACL 2024, pages 2922-2931. Association for Computational Linguisties, June 2024. doi: 10.18653/v https///////224..n/ggggg/// maddngs--nacl.1s. URL
org/ 1.2.4.findiig findings- naacl. 184/
[42]
Yifan Song, Weimin Xiong, Xiutian Zhao, Dawei Zhu, Wenhao Wu, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. Agentbank: Towards generalized LLM agents via fine-tuning 50000+ interaction trajectories. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguisties: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pages 2124-2141, Association for Computational Linguistics, 2024. URL Stthtt:::: / http//aalaanttt /aaph/ocaannnnoooooooooco org/2024// findings emnlp. 116.
[43] Timo Schick, Jane Dwivedi-Yu, Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Lan- guage models can teach themselves to use tools. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference tion Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 htcepaper_fieess Function calling at the edge. arXiv, -eee--bssstttteee Abstracct-Conferennc Conference html. arXiv. 2409. 00608.
Neural Informa-
16. 2023, 2023. URL http:: .ips..iip....................m.. www..ncippspppppoooo cc/paper files wilesspapeprs http//appp/22//aaaaeeee 1442222441190033320000006.03
[44] Lutfi Eren Erdogan, Nicholas Lee, Siddharth Jha, Sehoon Kim, Ryan Tabrizi, Suhong Moon, Coleman Hooper, Gopala Anumanchipalli, Kurt Keutzer, and Amir Gholami. Tinyagent:
2024. URL https:t/tts.iii... wtwg//...850..
15

---

[45] Graziano A. Manduzio, Federico A. Galatolo, Mario G. C. A. Cimino, Enzo Pasquale Scilingo, and Lorenzo Cominelli. Improving small-scale large language models function calling for reasoning tasks. arXiv, 2410. 2024. http:t // org/ 10. 48550/ arviv 2410. 18890.
[46] Ne Luo, Aryo Pradipta Gema, Xuanli He, Emile van Krieken, Pietro Lesci, and Pasquale Minervini. Self-training large language models for tool-use without demonstrations. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Findings of the Association for Computational Linguistics: NAACL 2025, Albuquerque, New Mexico, USA, April 29 May 4, 2025, pages 1253-1271. Association for Computational Linguistics, 2025. URL https: htttp://ddo.oor//0... 18653/v1/202... findiing.. wtwp://s-iiiinnnnnnnaaaaaa naaci. 69.
[47] Ibrahim Abdelaziz, Kinjal Basu, Mayank Agarwal, Sadhana Kumaravel, Matthew Stallone, Rameswar Panda, Yara Rizk, G. P. Shrivatsa Bhargav, Maxwell Crouse, R. Chulaka Gu- nasekara, Shajith Ikbal, Sachindra Joshi, Hima Karanam, Vineet Kumar, Asim Munawar, Sumit Neelam, Dinesh Raghu, Udit Sharma, Adriana Meza Soria, Dheeraj Sreedhar, Praveen Venkateswaran, Merve Unuvar, David Cox, Salim Roukos, Luis A. Lastras, and Pavan Kapa- nipathi. Granite-function calling model: Introducing function calling abilities via multi-task learning of granular tasks. In Franck Dernoncourt, Daniel Preotiuc-Pietro, and Anastasia Shimorina, editors, Proceedings of the 2024 Conference on Empirical Methods in Natu- ral Language Processing: EMNLP 2024 Industry Track, Miami, Florida, USA, Novem- ber 12-16, 2024, pages 1131-1139. Association for Computational Linguistics, 2024. URL attps://do1 http://o..o///o/////
org/ /0/ 18653/6633180 200003/v12222// emnlp- hepp-nndustt..ee .8..
[48] Anna Goldie, Azalia Mirhoseini, Hao Zhou, Irene Cai, and Christopher D. Manning. Synthetic data generation & multi-step rl for reasoning & tool use, 2025. URL https: org/ abs/2504. 047354404736..
[49] Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search- rl: Training iims to reason and leverage search engines with reinforcement learning. arXiv, 2503. 05536, 2025. URL https: / /doi. org/ 10. 48550/arXiv. 2503. 09516.
[50] Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models. arXiv, 2501 603.66666, 2025. URL http:: //tp::/dooio.oooo org/10. 1011000.00 488.00/ /ar5vv/ 2501 .03366..
[51] Edward Beeching, Shengyi Costa Huang, Albert Jiang, Jia Li, Benjamin Lipkin, Zihan Qina, Kashif Rasul, Ziju Shen, Roman Soletskyi, and Lewis Tunstall. Numinamath 7b tir. https: 7B- 2024.
/ //huggingface.cc ca
[52] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL id-EpOTtjVose
wetiieewweeeeeeeeeee wwt.eee//oooooo ffoum??
[53] Chengpeng Li, Mingfeng Xue, Zhenru Zhang, Jiaxi Yang, Beichen Zhang, Xiang Wang, Bowen Yu, Binyuan Hui, Junyang Lin, and Dayiheng Liu. START: talght reasoner with tools. arXiv, 2503.04625 2025. URL https: //doi. http://oooooogg/////
04625.
[54] Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. Toolrl: Reward all tool learning needs, 2025. URL https: 13958.
/ //arxiv. hrrg/abs/200
[55] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. CoRR, 0ts/22409.11222 2024. URL. https:/ org/ 10. 4014448555i...... arxiv. 240...240911101 12122.
[56) Salaheddin Alzubi, Creston Brooks, Purva Chiniya, Edoardo Contente, Chiara von Gerlach, Lucas Irwin, Yihan Jiang, Arda Kaz, Windsor Nguyen, Sewoong Oh, Himanshu Tyagi, and
16

---

Pramod Viswanath. Open deep search: Democratizing search with open-source reasoning agents. arXiv, 2503,20201, 2025. doi: 10.48550/ARXI.2253.22222 URL https://doi. 0Fg/10.485500 20150//arXiv.
[57] Joykirat Singh, Raghav Magazine, Yash Pandya, and Akshay Nambi. Agentic reasoning and tool integration for Iims via reinforcement learning, 2025, URL bttps://arxxe org. /bs///////// 2505. 01441.
[58] Junde Wu, Jiayuan Zhu, and Yuyuan Liu. Agentic reasoning: Reasoning Ilms with tools for the deep research. arXiv, 2502.04644, 2025. UtL.https:/ddoi www/oogg//////.oo 48050//r55arii 2502. 04644.
[59] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. arXiv, 1503.02531, 2015. URL htttttttp//riiii http://aarxi...orra/////////// wwrrg/abs/1503.. 02533.
[60] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[61] Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and Erik Kaunismäki. 'smolagents': : a smol library to build great agentic systems. https://githh:: com com/huggingfaace//sie smo smolagens,, 2025.
[62] Ke Yang, Jiateng Liu, John Wu, Chaoqi Yang. Yi R. Fung, Sha Li, Zixuan Huang. Xu Cao, Xingyao Wang, Yiquan Wang, Heng Ji, and Chengxiang Zhai. If LLM is the wizard. then code is the wand: A survey on how code empowers large language models serve as intelligent agents. CoRR, abs/2401. 1.0001.001222 2, 2024. URL https: http://ooooo///////////////
org/ 10. 48550/ /arvvv. 2401. 00812.
[63] Kushal Jain, Moritz Miller, Niket Tandon, and Kumar Shridhar. First-step advantage: Importance of starting right in multi-step math reasoning, 2024. URL 07945.
[64] Ke Ji, Jiahao Xu, Tian Liang, Qiuzhi Liu, Zhiwei He, Xingyu Chen, Xiaoyuan Liu, Zhijie Wang, Junying Chen, Benyou Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. The first few tokens all you need: An efficient and effective unsupervised prefix fine-tuning method for reasoning models. Xiv,
2025. URL https: //doi. org/10. 48550/arXiv 2503 02875.
[65] Zicheng Lin, Tian Liang, Jiahao Xu, Xing Wang, Ruilin Luo, Chufan Shi, Siheng Li, Yujiu Yang, and Zhaopeng Tu. Critical tokens matter: Token-level contrastive estimation enhances Ilm's reasoning capability. CoRR, 2411.19943, 2024. URL https: :/do:i:///.... http:///ooooo//oo///// org/10 /0111004444 / arXiv 2411. 19943.
[66] Seanie Lee, Haebin Scong, Dong Bok Lee, Minki Kang, Xiaoyin Chen, Dominik Wagner, Yoshua Bengio, Juho Lee, and Sung Ju Hwang. Harmaug: Effective data augmentation for knowledge distillation of safety guard models. International Conference on Learning Representations (ICLR), 2025.
[67] Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma. Subhrajit Roy, Ahmad Beirami, Prateck Mittal, and Peter Henderson. Safety alignment should be made more than just few tokens deep. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openrevee/
wwt//eeuu//iiiit htt.:rrrrrnndddddtgggggo 6Mxhg9PtDE.
[68] Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does LLM safety training fail? In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference Neurul Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023, URL http:://tiit //pppers nips wwtppppp.ppceeeeeeee.. 2023
cttpp/pprffflleee///eeepeeeeeeee falerrfielepppppe /pperr/ htt.:aaaabbbooo00-o-orttttt..tee Abstract- Confer Wwwsssrrcc--oofeeenuttee html.
[69] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URI, https: //tp://v/ee.......//// net/foruu?Pd:rygGByyyy
17

---

[70] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le. Ed H. Chi. Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023.
[71] Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. MINT: evaluating llms in multi-turn interaction with tools and language feedback. In The Twelfth Internationaa Conference on Learning Representations. ICLR 2024, Vienna, Austria. May 7-11, 2024. OpenReview.net, 2024. URL /ttttps://pperevvvvie nwt sett/ffrmmiddjjip jp3gddru332
[72] Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li. Erie P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, edi- tors, Advances in Neural Information Processing Systems 36: Anmual Conference on Neu- ral Information Processing Systems 2023, NeurlPS 2023, New Orleans, IA,, USA, December wwwes//appprreeeee hatps/appee///ae 222222222aaas
10 - 16, 2023, 2023. URL http:: wwwpaaprrrrrsip....ee.pp.pspsssm cc/ wc/paper__feeeeeeeeeeeee firess/iee////eeeeeeer 11111118888333888888888888588883 1tw.-0fff4411-----aaaa--- Abstract Datasets_ aad_Bennhmarke html.
[73] Patrick S. H. Lewis. Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin. Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sehas- tian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020. URL http:::///////t htttps://prroeeednggg wwwpiiiinnnnnnuuiiieee nwurips. cc/ pttppppppp///aaaa///eepp 1b49333022061880111222246468
Abstract html.
[74] Liang Wang, Nan Yang. Xiaolong Huang. Binxing Jiao, Linjun Yang. Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv, 2212.03533, 2022. URL https: org/ 48550/arXiv http:///.rrr...44/0///////1
03533.
[75] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https: htttps://opperevei/ net//oorum2.d-nZeeeeer
[76] Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen, Jun Kun Chen, Weizhu Chen, Yen-Chun Chen, Yi-ling Chen, Qi Dai, Xiyang Dai, Ruchao Fan, Mei Gao, Min Gao, Amit Garg, Abhishek Goswami, Junheng Hao, Amr Hendy, Yuxuan Hu, Xin Jin, Mahmoud Khademi, Dongwoo Kim, Young Jin Kim, Gina Lee, Jinyu Li, Yunsheng Li, Chen Liang. Xihui Lin, Zeqi Lin, Mengchen Liu, Yang Liu, Gilsinia Lopez, Chong Luo, Piyush Madan, Vadim Mazalov, Arindam Mitra, Ali Mousavi, Anh Nguyen, Jing Pan, Daniel Perez-Becker, Jacob Platin, Thomas Portet, Kai Qiu, Bo Ren, Liliang Ren, Sambuddha Roy, Ning Shang, Yelong Shen, Saksham Singhal, Subhojit Som, Xia Song, Teryana Sych, Praneetha Vaddamanu, Shuohang Wang. Yiming Wang. Zhenghao Wang, Haibin Wu, Haoran Xu, Weijian Xu, Yifan Yang. Ziyi Yang. Donghan Yu, Ishmam Zabir, Jianwen Zhang, Li Lyna Zhang, Yunan Zhang, and Xiren Zhou. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture--ffllore arXiv, 2503.01743, 2025. URL org/10 10/ 48550/ 48550/arXiv 2503. 01743.
https://doi.
[77] Binyuan Hui, Jian Yang. Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, An Yang, Rui Men, Fei Huang, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. Qwen2.5-coder technical report. arXiv, 2409.12188 2024. 2409. 12186.
URL https: //doi. org/ 111.1..88855 00-448855///00
[78] Sanjiban Choudhury. Process reward models for LLM agents: Practical framework and direc- tions. arXiv, 1002, 2025. URL attps:///oi. org/10. 48550/ /wt.i//r/vvi.... 2502. 10325.
[79] Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Lun- Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024,
18

---

net/ Bangkok, Thailand. August 11-16, 2024, pages 9426-9439 Association for Computational Linguisties, 2024. URL https://doi. org/10. 18653/v1/2024. acl-long. 510.
[80] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv, 2402.03300, 2024. URL https://dos org/10. 48550/ arxiv. 2402.03300.
[81] Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean-Bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Eti- enne Pot, Ivo Penchev, Gaøl Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Róbert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Jain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan-Thore oun-hhosteen Peter, Danila Sinopalnikov, Surya Bhupatiraju. Rishabh Agarwal, Mchran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Anto- nia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Di- vyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimczak-Pluciinkka Harman Singh, Harsh Mehta, Harshal Tushar Lchri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, and Ivan Nardini. Gemma technical report. arXiv, 35003197866 2025. URL https: //oii. org/ 10. 48550/ arxiv 2503. 19786.
[82] Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew J. Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL forum??d=010x0Ycoooo
[83] Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scal- able real- woaldw web interaction with grounded language agents. In Sanmi Koyejo, S. Mo- hamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neu- ral Information Processing Systems 35: Annual Conference on Neural Information Pro- cessing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 Decem- wccppaape_fiiiiieeeeeeeeeeeeeeee
ber 9, 2022, 2022. URL http:://aapeeee Awstract-Coottteeeeeeeeeae www.fflllppppeeeee haall///apeer2222//aaaae html.
[84] Anthropic. Introducing the model context protocol. https: model-context- www.cootttt---oooooooooooo prw.toolotooooo. 2024.
[85] Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingehen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma. Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, and et al. Openhands: An open platform for AI software developers as generalist agents. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL BRLhttps: stttps://openreviiee net http://www......oooo/////ooaaaaa
19

---

A Limitations
In addition the discussion in Section 8, we outline here several additional limitations of our study.
First, our experiments are limited to the Qwen2.5 model series [1]. While we expect our proposed approach to generalize across model families, we have not validated its effectiveness
other widely- used language models such as LLaMA [11] or Gemma [81]. Extending our study to these models would strengthen the generality of our findings and remains important direction for future work.
Second, we only distill from a single teacher model (Qwen2.5-32B) Using stronger or larger teacher models--paticularly proprietary closed-source models like GPT-4 [12]-may lead to further perfor- mance gains in student agents. However, such experiments were not feasible due to computational and budget constraints.
Third, we do not investigate the effect of the number of teacher trajectories per question on student performance, which has been shown to be an important factor in prior CoT distillation research [16, 19]. Exploring this variable could offer further insights into how to optimize agent distillation.
Lastly, our current work focuses exclusively on agents that utilize retrieval and code execution tools to solve real- world problems that the general LLM can solve without tools. Other agent applications- such as embodied agents [82] or web-based agents (83]-remain unexplored. Future research could extend agent distillation these broader settings, leveraging tool-augmented environments such as web browsers, simulators, or desktop interfaces. In particular, integration with frameworks like the Model Context Protocol (MCP) [84] could further enhance the capabilities of small agents across diverse real-world tasks. Furthermore, ensuring safety during code execution is crucial, as unsafe operations generated by small language model agents can be irreversible harmful. A promising direction is to apply safery-tuned decoding to reduce the likelihood of generating unsafe code or to execute code within sandboxed environments such Docker or E2B [85].
B Broader impacts
This work contributes toward the development of small language agents capable of running on local devices, enabling functional on-device AI that can retrieve information from external knowledge sources (including the web) and perform code-based action to complete complex tasks.
On the positive side, this advancement promotes more accessible and inclusive AI by lowering the hardware and computational barriers for deployment. It opens opportunities for broader adoption privacy-sensitive domains, such healthcare, where data
of AI agents in essource-constriigg locality and privacy are critical.
However, there are potential risks. Since our distilled agents are capable of retrieving web information and executing code, they could be misused to automate malicious behaviors, such as generating harmful scripts or launching unauthorized attacks. Addressing these concerns will require the integration of robust safeguards, including behavior monitoring, tool-use restrictions, and secure deployment practices. We highlight this as an important avenue for future research and responsible development.
C Implementation details
Prompts and agent framework. For CoT prompt, we use the prompt in Prompt D.1 for both math and factual reasoning. For agent prompt, we use the prompt from smolagents library [61]. We present the part of prompt in Prompt D.2.
As an agent framework, we use the CodeAct [21] implemented in smolagents. We only include the retriever for wikipedia as a tool with the name of
For student model, we use the same prompt for CoT reasoning. For agent, we only remove few-shot demonstrations as it is no longer needed after fine-tuning
Training dataset details. We use 1000 HotPotQA [2] and 2000 MATH [6] examples for training. Specifically, we only use 1000 hard examples from HotPorQA and 1000 level 2-3 examples, 1000 level 4-5 examples from MATH dataset. We prompt LLM to generate trajectories for both CoT
20

---

Table 5: Comparison of CoT and Agent approaches Sween..5-3BB-Insteee training dataset.
denotes the first-thougtt prefix. Hard denotes 4-5 and medium denotes level questions.
Model
HotporQA
MATH (hard)
MATH (medium)
CoT
Qwen2.5-32B-n Instruct
40.9
71.1
89.8
Swen2.5-32B-Insteee Dwwen2.5-32-IInstmee
59.3
58.4
78.4
Agent
PTP
60.8
67.1
83.4
and agent and filter out wrong trajectories based the correctness of predicted answer. After filtering, we use approximately 2,000 trajectories to train the small models. The exact number varies depending on the performance of the teacher models on the training dataset, which we present details in Appendix D.I.
D Additional analysis
D.1 Teacher model performance training dataset
In Section 4, we propose that the first-thought prefix improves teacher performance on hard math problems. To support this, we present teacher model results the training set in Table We observe that the LLM agent outperforms a chain-of-thought (CoT) prompted LLM in factual reasoning, as the LLM relies heavily on prompting use tools effectively- and proper tool use contributes significantly to performance. However, the performance of the LLM agent on math tasks drops considerably, especially harder (level 4-5) problems.
In such cases, adding the first-thought prefix helps recover some of the lost performance, as discussed in Section 4. These results suggest that simply prepending the first CoT step the agent's reasoning improves its capabilities, which in turn benefits distillation, as shown in Table
D.2 Failure case analysis of agent the math subcategory
In example D.I, we present failure case of the distilled 3B agent on a level 2 precalculus problem. In this instance, the generated code produces decimal result, which is not the correct form for an answer expected in radians. Although the agent attempts a conversion in its reasoning, it ultimately produces an incorrect radian value.
Examples D.2 and D.3 involve more challenging level 4 precalculus problems. In Example D.2, for instance, the agent makes conceptual error in its reasoning by misidentifying the appropriate range for the angle 0.
These examples suggest that the agent struggles particularly with problems requiring analytic reasoning-sune as understanding the properties of trigonometric functions-rather than straightfor- ward computation.
D.3 Deeper analysis on the first-thought prefix
Effects mathematical reasoning. As discussed in Section 4, the inclusion of first-thought prefix (FTP) influences the initial reasoning patterns of the agent. this section, we analyze how this prefix affects student agents distilled from trajectories both with and without the FTE, using representative examples.
In examples D.4 and D.5, drawn from the MATH500 dataset, compare the reasoning approaches of distilled 3B agents with and without the FTP. Without the prefix (Example D.4), the agent's initial reasoning begins with a descriptive analysis, e,g. "The problem is asking...." focusing understanding the question. In contrast, with the prefix (Example D.5), the agent begins with a goal- oriented plan, e.g., "To find the smallest positive real number.... which mirrors a chain-of-thougge (CoT) strategy.
This shift illustrates that the FTP nudges the agent toward more proactive and structured reasoning style, which might be beneficial in domains requiring multi-step reasoning (e.g., challenging math problems).
21

---

Table Effect of temperature math reasoning performance after agent distillation. The experiments done
with Qwen2. 5-1 1eeen..51..B-Instaeeee 58-Instruct with both FTP SAG. Bold numbers indicate best results each column.
Temperature
MATH500
GSM-Hard
AIME
OlymMATH
Avg (Math)
0.2
48.0
60.2
7.8
3.5
29.87
0.4
50.6
60.6
6.7
4.5
30.59
0.6
50.8
61.8
4.4
4.5
30.39
0.8
52.4
61.8
4.4
3.5
30.54
1.0
51.0
63.8
6.7
3.5
31.24
Table 7: Average and standard deviation across 5 different seeds in inference for agent distilled Qwen2.5 5 model scales AIME with both PTP and SAG.
Model
Avg
Std
0.5B
2.00
0.93
1.5B
6.23
0.61
3B
14.44
1.36
Effects in factual reasoning. As shown in Figure 8, the use of the first-thought prefix (FTP) consistently reduces the number of retrieval tool calls made by distilled agents. To better understand this phenomenon, we include illustrative examples from the Bamboogle dataset.
Examples D.6 and D.7 demonstrate knowledge internally rather than retrieving it. This question requires identifying the founder of geometry, the city associated with that individual, and the founder of that city.
where the causes the distilled agent generate factual
In Example D.6, the agent (with FTP) directly generates the statement "The founder of geometry, Euclid" without making retrieval call. In contrast, in Example D.7, the agent (without FTP) uses the retrieval tool to search for the founder geometry, which reduces the risk of hallucination.
This pattern helps explain the behavior observed in Figure 8: while FTP can reduce the number of tool calls, may also increase likelihood of factual errors due hallucination, as the agent relies more internally generated knowledge.
D.4 Deeper analysis the self-consistent action generation
Temperature ablation To examine the effect of sampling temperature in self-consistent action generation (SAG), we evaluate the distilled Qwen2. 5-1.5B- InSIrrtuct student model across temper- atures of 0.2, 0.4, 0.6, 0.8, and 1.0 on MATH500, GSM-Hard, AIME, and OlymMATH. As shown Table 6, performance remains relatively stable across all settings, with variations within roughly 2%. While higher temperatures (e.g., T = 1.0) slightly improve average accuracy by increasing action diversity, lower values (e.g., T 0.4) also yield comparably strong results. These findings suggest that SAG is insensitive to the precise temperature choice, and we adopt T = 0.4 in the main experiments as a balanced configuration between diversity and reliability.
Variance analysis Since we stochastically sample trajectories from the model SAG, randomness can introduce variation in evaluation results. This effect can be particularly noticeable for AIME, which contains only 90 questions and thus can exhibit higher variance due its small size. To verify that our method yields consistent performance regardless of randomness (e.g., random seed), we conduct inference five times with different random seeds on AIME, As shown Table 7, the observed variance is small, corresponding to only one or two questions of difference across runs.
D.5
Full fine-tuning vs. LoRA
All of our main experiments employ LoRA [75], owing to its low memory footprint and ease of deployment through compact adapter weights. To assess whether full fine-tuning can offer additional benefits in terms of performance, we fine-tune the Qwen2. 2.....511... IB-Instruct model for two epochs with a learning rate of x 10-5 fixing other hyperparameters unchanged compared to LoRA. As shown in Table 8, full fine-tuning yields lower average performance than LoRA-based training. While
22

---

Table 8: Comparison between LoRA and full fine-tuning (FT) on
Method
Hotpot.
MATH
MuSiQue
Bamb.
2Wiki.
GSM-H.
AIME
Olym.
Avg.
Agent Distill (LoRA)
43.6
46.4
8.0
30.4
32.6
60.6
7.78
3.5
29.11
Agent Distill (Full FT)
6.2
20.0
52.0
26.24
further hyperparameter tuning may improve the results, this trend suggests that full fine-tuning is more prone to overfitting and generalizes less effectively, making parameter-efficiet adaptation preferable for agent distillation.
23

---

Prompt D.1: Prompt for Slain-of-Thougee Reasoning
You are expert assistant who can answer the given question accurately and provide clear reasoning.
When answering questions, follow these guidelines:
1. Provide a clear and structured reasoning first
Follow up with final answer, must in the <answer> </answer> tag. For example, <answer> XXX </answer>.
The answer must be succinet and final. For math problems, return the answer using LaTeX in the \boxed format.
4. If the question requires multiple steps or facts, break down your reasoning accordingly
5. Be precise and factual in your responses
If you't re unsure about something, acknowledge the uncertainty
Now, please answer the following question:
Prompt D.2: Prompt for Agent
You are an expert assistant who can solve any task using code blobs. You will be given a task solve best you can. To do so, you have been given access to a list of tools: these tools are basically Python functions which you can call with code. To solve the task, you must plan forward to proceed in a series of steps, in a cycle of Thought:, Code:, and Observation: sequences.
At each step, in the Thought: sequence, you should first explain your reasoning towards solving the task and the tools that you want to use.
Then in the Code: sequence, you should write the code in simple Python. The code sequence must end with <end_ _ode> sequence.
During each intermediate step, you use print () to whatever important infor- mation you will then need.
These print outputs will then appear in the Observation: field, which will be available as input for the next step.
answer tool. In the end you have return final answer using the final_answer
For math problems, not specified, always return LaTex format as the final answer. (ellipsis).... ...
Here are the rules you should always follow to solve your task:
Always provide Thought: sequence, and a Code:"' "** py sequence ending with '* <end_code> sequence, else you will fail.
Use only variables that you have defined!
3. Always use the right arguments for the tools. DO NOT pass the arguments as diet as in answer wiki ('query': "What the place where James Bond lives?"), but use the arguments directly as in answer wiki (query="Whal is the place where James Bond lives?").
4. Take care not chain too many sequential tool calls in the same code block, espe- cially when the output format is unpredictable. For instance, a call to search has an its output
unpredictable return format, so do not have another tool call that depends the same block: rather output results with print() to use them in the next block.
5. Call tool only when needed, and never re-do tool call that you previously did with the exact same parameters.
6. Don't name any variable with the same variable final_ final_answel answer.
tool: for instance don't ton't name
Never create any notional variables in code, having these in your logs will derail you from the true variables.
(ellipsis)...

---

Example D.I: Failure case of Agent in Precalculus (level 2)
Problem: Compute arcsin (-
Express your answer radians.
Agent:
Thought: To compute aresin (-2), need find the angle such that sin(O) and & lies within the range of the arcsine function, which is [- 3 7].
Code:
import math
Calculate the arcsine of -1/2 theta = math asin print (theta)
Observation: 00$235987759982989
Thought: The output from the code snippet is the value of 0 in radians. However, the task requires the answer be LaTeX format and as an exact value, not decimal approximation. The value of 0 is bn which is the exact value of the aresine of within the range of the aresine function.
Code:
final answer (* ("Noxxed\\trcc((5)))))))
5
Observation:
(wrong)
CoT:
To compute aresin (-1), we need to find the angle @ such that sin(0) and lies within the range of the arcsine function, which is [- 7]
Recall that sin(0) - corresponds the sine of an angle in the unit circle where the
y-coordinate is -
2. The sine function is negative in the third and fourth quadrants. However, since the range of the aresine function is - 1.1 we are only interested the fourth quadrant for this problem.
3. The reference angle for sin = 1 is T Therefore, the angle in the fourth quadrant that has a sine value of is 8. Thus, the angle 0 such that sin( - and 0 |- is 6'
(correct)
25

---

Example D.2: Failure case of Agent in Precalculus (level 4) part
Problem:
Let [(ar)) = (arccos x² + (aresin xy Find the range of I ((x functions are
radians.
Agent:
Thought:
To find the range of the function f(x) = (arccos x)² + (arcsin x)?, we start by using
the identity that relates the arccosine and arcsine functions:
π 2
arccos arcsin
Let 0 a. Then aresin it - -0. Substituting these into the function, we get:
f(x) - 0² 2 -
Expanding the expression, we have:
f(x) = 02
2
7 8² 20²
Tb +
This is a quadratic function in terms of 0. To find the range of f(x), we need to determine the range of 9 and then find the minimum value the quadratic function over this range. The range of 0 is [0, 211 We can find the minimum value of the quadratic function f
20² x + ( by completing the square or using the vertex formula for a quadratic function
b. Let's find the minimum value of f(() using the vertex formula. ax + br which is = 2u
Code:
from вympy import aymbols, diff, solve, pi
Define the variable and function theta ('thetaa)))))
f_theta = 2*theta**2 pi*theta (pi/2)**2
Find the derivative of the function
f_prime_theta tietta, theta)
Solve for the critical points critical.points solve solve (f..mmff.m_tteeta. theta) print(criitical.pints
11
12
Observation: [T/4]
Thought: The critical point is 0 = 4. We can now evaluate the function f(0) at this point to find the minimum value of the function. Let's evaluate (0) at 0 4
Code:
Evaluate the function at the critical point theta_critical pi
f_theta_critien f_theta, subs (theta, theta_critical) (f herffthheaa.rriiiiicee
Observation: 2/88
Thought: The minimum value of the function f(0) 0 T is Since the function is quadratic function that opens upwards (the coefficient of 02 is positive), the minimum value is the only value in the range of f(x). Therefore, the range of f(x) is .
Let's use the final answer tool to provide the final answer in LaTeX format. Code:
final answer ("Nboxed(Vleft. (trac((pir2H00 Vinfty \right))")
22
Observation:
(wrong)
26

---

Example D.3: Failure case of Agent in Precalculus (level 4) part
Let [(ar)) = (arccos x² + (aresin v) Find the range of I (xx functions are in radians.
Problem:
CoT:
(arccos x)² + (arcsin x)?, we start by using the relationship between arccos and arcsin Recall that for any [-1,1], 1],
find the range of the function f(x)
π arccos x + aresin x 2
Let = arccos æ. Then aresin = = 0. Substituting these into the function f(x), we get
f(x) 0²
( -0
Expanding the expression, we have
f(x) = 02 + 2 π -
8² +
π 2
T0 + 02 202 T0
n²
This is a quadratic function in terms of 0. To find the range of f(z), we need to determine the range of 0 as z varies from -1 to 1. Since arccos x, 9 ranges from 0 æ.. We need find
the minimum and maximum values of the quadratic function g = 202 T + over the interval [0, x].
The vertex of the parabola g(0) = 20 TO + = occurs at
b 2a
5 Evaluating g()) at
the vertex and at the endpoints of the interval [0, m]:
1. At = 0:
2
n²
s 2(0) =(0)
2. At 0
g(T) 2(xT) 3 (TT) +
x²
2m
2
2
π
5T²
4
3. At :t
π
1 2
n
7
7
2
n
2
π
4
16
4
4
Therefore, the range of f(x) is The minimum value of g(0) is and the maximum value is
x²
4
2
5 2
8
π² 5x
Thus, the range of (x) is
(correct)
8
4
27

---

Example D.4: Comparison of first-thought prefix
Problem: Find the smallest positive real number C' for which
16:
-2 1 S Clivll
for all two-dimensional vectors v.
Note that for two-dimensional vector a, |||| is the magnitude of a.
Small Agent distilled from trajectories without FTP:
Thought: The problem is asking for the smallest positive real number for which the inequality holds for all two-dimensional vectors This is problem of finding the spectral radius of the matrix. The spectral radius matrix is the largest absolute value of its eigenvalues. I will first find the eigenvalues of the matrix, then find the largest absolute value of the eigenvalues, and that will be the value of C. Code:
import numpy as
Define the matrix np. arrarray ([[2, 3], [o, -2]])
# Find the eigenvalues
eigenvalues np. eigvals
Find the largest absolute value of the eigenvalues C max (abs (ax(ab(eeiggn)) (eigenvalues))
print (C)
Observation:
2.0
Thought: The code has executed and the value has been found. Now will return the final using the final_answer answer tool.
Code:
final_ answer (") (2)
Observation:
2
(wrong)
28

---

Example D.5: Comparison of first-thought prefix (cont.)
Problem: Find the smallest positive real number C' for which
16: -2 1 S Clivll
for all two-dimensional vectors v.
Note that for two-dimensional vector a, |a|| is the magnitude of a.
Small Agent distilled from trajectories with FTE: Thought: To find the smallest positive real number C' for which the incquality
( - )
Cllvll
holds for all two-dimensional vectors V, we need to analyze the norm of the matrix-vecteo product. The norm of the matrix-vector product can be computed as the maximum singular value of the matrix. The singular values of matrix are the square roots of the eigenvalues of the matrix multiplied by its transpose. For 2x2 matrix A roots of the eigenvalues of A"A.
d) the singular values are the square
2
3
Let's compute the singular values of the matrix A =
0
-2
Code:
import numpy as np
Define the matrix A np. ap.array ([[2, 3], [0, -2]1)
Compute the singular values of the matrix singular_values np linalg.svd compute_u-=aase)
Observation:
1.]
The singular values of the matrix A (
3
Thought:
-2 are and 1. The maximum singular value 4. Therefore, the smallest positive real number C' for which the inequality holds for all two-dimensional vectors is the maximum singular value.
Code:
The maximum singular value is smallest max (singular_ final_answed (C)
Observation: 4 (correct)
29

---

Example D.6: Illustrative case of first-thought prefix in factual reasoning
Problem:
Who founded the city where the founder of geometry lived?
Small Agent distilled from trajectories with FTP: Thought:
To answer this question, we need to identify the city where the founder of geometry, Fuclid, lived and then determine who founded that city.
Code:
soelid_birthpacce web_search (query- where did Euclid live") print (f "Eccli lived in bouclid.birtppaao).
Observation:
Euclid lived
(...)
"Euclid, Ohio"
Euclid, Ohio Euclid is city in Cuyahoga County, Ohio, United States. is an inner ring suburb of Cleveland. As of the 2010 census, the city had total population of 48,920. In 2009, Euclid celebrated its bicentennial. The city named after Fuclid, the Greek mathematician. The city was settled in 1796 and became a township in 1809. Euclid then became a village in 1903 and a city in 1930, during the Great Depression. The city (then village) was the primary appealing party in the landmark case ""Village of Euclid v. Ambler Realty Co. "" Euclid is located (...)
Thought: From the search results, we can see that Euclid, Ohio was named after the Greek mathematician Euclid. Now, we need to find out who founded the city of Euclid, Ohio. Code:
founder lander_of_euclid = web_searce (query= " who founded the city of Buclid, Ohio") print (f The city of Euclid, Ohio founded by { founder_of fo
Observation:
The city of Euclid, Ohio founded by
(...)
"South Euclid, Ohio"
treaty with the Iroquois confederation in 1796 by the Connecticut Land Company. 1797, Moses Cleaveland named the area east of the Cuyahoga River Euclid, after the Greek mathe- matician and patron saint"" "" of surveyors. Euclid Township was officially formed in 1809. 1828, Euclid Township divided into nine districts, with South Euclid becoming district two. (...)
Thought: The search results indicate that the city of Euclid, Ohio was settled in 1796 by Moses Cleaveland, who was trained in mathematics and named the township after the Greek mathe- matician Euclid. Therefore, Moses Cleaveland founded the city where the founder of geometry, Fuclid, lived. Code:
final_ answer (MMose Cleaveland")
Observation:
Moses Cleaveland (wrong)
30

---

Example D.7: Illustrative case of first-thought prefix in factual reasoning (cont.)
Problem: Who founded the city where the founder of geometry lived?
Small Agent distilled from trajectories without FTP:
Thought: need to find out who founded the city where the founder of geometry lived. I will first find out who the founder of geometry is, then find out where he lived, and finally find who founded the city where he lived. will use the tool 'web_search to get this information. Code:
founder of geooetryy = web_ search (query=" of geometry") print Founder of geometry: -founder_off "f.gmmettry))""
Observation:
Founder of geometry:
(...) Euclid
Euclid Euclid (; ""Eukleidēs""; fl. 300 BC), sometimes given the name Euclid of Alexandria to distinguish him from Euclides of Megara, was a Greek mathematician, often referred to the ""founder of geometry"" or the ""father of geometry"". He was active in Alexandria during the reign of Ptolemy (323-28- 283 BC). His ""Elements" " is (...)
Thought: From the search results, I can that the founder of geometry is Euclid. Now I need to find out where lived and who founded the city where he lived. I will use the tool aweb_searchh again to get this information. Code:
euclid lived = web aearch where did Euclid live") print (f " WWere did Euclid live: [euclid_lived)"")
Observation:
Where did Euclid live:
(...) Euclid
A detailed biography of Euclid is given by Arabian authors, mentioning, for example, birth town Tyre. This biography is generally believed be fictitious. came from Alexandria, he would have known the Serapeum of Alexandria, and the Library of Alexandria, and may have worked there during his time. (...)
Thought: From the search results, I can that Euclid lived in Alexandria, Egypt. Now need to find out who founded the city where Euclid lived. I will use the tool 'web_search" again to get this information. Code:
Sander_of_alexanee
web_search (query="uho founded Alexandria"
print WWo founded Alexandria: {founder _farrroof_aleeee of_.aexanddiia)"))
Observation:
Who founded Alexandria:
(...) Alexandria
Alexandria believed have been founded by Alexander the Great in April 331 BC ("" ("Allxandreiaa"". Alexander's chief architect for the project was Dinocrates. Alexandria was
intended to supersede Naucratis as a Hellenistic center in Egypt, (...)
Thought: From the search results, I can that Alexandria founded by Alexander the Great. Therefore, the founder of the city where the founder of geometry lived is Alexander the Great. will now provide the final answer using the 'final answer' tool. Code:
final answer finwer
(" (eAeeanderr the Great")
Observation:
Alexander the Great (correct)
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Section 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section 8 and Appendix A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
32

---

Justification:
Our paper does not include any theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section 4 and Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
33

---

Answer: [Yes]
Justification:
We include codes for reproducing experiments as supplementary files.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips wwtppbccc//aaeeccccccccccccc httpe/guuuieee/eeuueuaeaee /at////ooommsssseeeeee ssisssssicyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
See Section
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due extensive computational resources of language model experiments, only present the experimental results after the single run.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
34

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http///eeerppeeeeeeeeeee httt::::harrppp.cppublbi.l. wttp:/thhiccceellleeeeeeeeeeeeee ines?
Answer:
[Yes]
Justification:
The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Appendix B.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
35

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release data or models as this work aims the method for distillation.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper cites all datasets and models used in Section 5.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
36

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/mmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not include any crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not include any potential risks incurred by study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
37

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: The paper describes the usage of LLMs throughout the paper including Sec-
tion 5.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: : /ttp//nurspp////// / 2LLM/L for what should or should be described.
38


Reviews:
[{"review_id": "vPMNYg6wIf", "reviewer": "Reviewer_qN16", "summary": "This paper introduces Agent Distillation, a method to transfer the reasoning and tool-use capabilities of LLMs to sLMs. The approach leverages LLM-generated trajectories of reason-act-observe cycles, incorporating two key techniques: First-Thought Prefix (FTP) to enhance the quality of the teacher LLM's trajectories, and Self-Consistent Action Generation (SAG) to improve the robustness of the student sLMs at test time.  Experiments on factual and math reasoning tasks show that agent-distilled sLMs outperform larger traditional CoT-distilled models.", "questions": "1. In describing Self-Consistent Action Generation (SAG), you state that you sample with a \"high temperature\" to encourage diversity, yet the value used is 0.4. In many contexts, this is considered a relatively low temperature that promotes less randomness. Could you clarify this choice and provide any ablation studies you performed on the effect of temperature on SAG's performance?\n2. Could you provide more detail on the evaluation protocol? Specifically, were the reported scores, especially for benchmarks with small size like AIME, averaged over multiple runs with different random seeds to ensure statistical significance?\n3. For the student models, you chose to use LoRA for fine-tuning. Given the relatively small size of both the models and the datasets, what was the rationale for using LoRA instead of full fine-tuning? Did you experiment with full fine-tuning, and if so, how did the results compare?\n4. An interesting trend appears on the MATH500 benchmark, where standard CoT prompting outperforms your distillation method for the 3B and 7B models, reversing the trend seen with the 0.5B and 1.5B models. Do you have a hypothesis for this scale-dependent performance difference?\n5. The methodology states that you \"filter out any sequences that result in parsing or execution errors\" (lines 160-161). Logically, this filtering should result in a zero error rate for the final selected action. However, the results in Figure 6 show that SAG still produces a non-zero error rate. Could you clarify why the error rate is non-zero, or how is it calculated?", "limitations": "● The effectiveness of the \"first-thought prefix\" (FTP) is only validated on a single 32B instruction-tuned teacher model. It is not clear whether this strategy is broadly applicable to other models or families, particularly Large Reasoning Models (LRMs) like DeepSeek-R1 or OpenAI-o1.\n● The paper's core technical contributions, FTP and SAG, are better described as effective applications of existing ideas (prompt engineering and self-consistency decoding, respectively) rather than fundamentally new methods. This limits the paper's originality.\n● The evaluation lacks details on its statistical robustness. The authors do not specify if results were averaged over multiple runs with different random seeds. This is a significant limitation, particularly for benchmarks with small test sets like AIME, where performance can have high variance and single-run scores are not reliable indicators of true performance.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Agent Distillation offers a practical and effective pathway for transferring the agentic capabilities of LLMs into smaller, more efficient sLMs. This presents a valuable contribution towards developing powerful on-device AI agents that can execute complex tasks without the high computational costs associated with LLMs.\n2. The paper is well-structured and the experiments are comprehensive. The authors evaluate their method on a diverse set of eight benchmarks, covering both in-domain and out-of-domain generalization for factual and mathematical reasoning. \n\nWeaknesses:\n1. While their application is effective, the core methods are derivative of existing work. The first-thought prefix (FTP) is a well-executed form of prompt engineering, designed to elicit a specific reasoning format. The self-consistent action generation (SAG) is a direct application of the well-established Self-Consistency decoding strategy. \n2. The paper proposes the first-thought prefix (FTP) to improve teacher quality but only validates it on a single model (Qwen2.5-32B-Instruct). It remains a question whether this specific prompting strategy is effective for other, more powerful reasoning models like DeepSeek-R1 or the OpenAI-o1, which may have different internal reasoning structures.\n3. The experimental analysis, while broad, lacks depth in key areas. The individual contributions of FTP and SAG are not clearly disentangled, making it difficult to quantify their separate impacts on the final performance. Key hyperparameters, such as the sampling temperature for SAG, are presented without ablation studies to justify their chosen values.", "was_revised": true, "final_justification": "The new AIME, multiple runs and full fintuning experiment have been further conducted to validate the effectiveness of the paper method. The novelty problem although still build on some existing methods, but has also certain difference.", "rebuttal": "We thank the reviewer for their valuable and insightful feedback.\nWe are glad to hear that the reviewer found our work (1) offers a practical and effective pathway for transferring the agentic capabilities, (2) presents a valuable contribution towards developing powerful on-device agents, (3) is well-structured and the experiments are comprehensive.\n\nTo address your concerns, we have clarified our contribution in terms of methods, run additional suggested experiments regarding ablation studies on SAG and multiple runs on AIME.\n\n---\n\n> W1. While their application is effective, the core methods are derivative of existing work. The first-thought prefix (FTP) is a well-executed form of prompt engineering, designed to elicit a specific reasoning format. The self-consistent action generation (SAG) is a direct application of the well-established Self-Consistency decoding strategy.\n\nWe acknowledge that both the First-Thought Prefix (FTP) and Self-Consistent Action Generation (SAG) are inspired by prior work. However, we would like to clarify that our contribution lies in **tailoring and validating these ideas in the novel context of agent distillation.**\n\n- FTP addresses the performance drop of language model agents in reasoning tasks. We find **prepending the first-thought from CoT in generating agent trajectories improves the performance of LLM agent by +5.07 (Table 4) as well as downstream performance of distilled agent (1.5B) by +1.05 (Table 2).** To our knowledge, this problem has not been identified or addressed in prior work.\n- For SAG, while it is motivated by self-consistency decoding, **our work is the first to adapt and evaluate it in the context of action generation for interactive language agents.** Unlike traditional uses of self-consistency that rely on access to ground-truth answers or additional LLM as a judge [1], we leverage majority voting over **observations from the environment** without external supervision. Importantly, this method remains effective even for the smallest 0.5B model evidenced by +1.91 performance improvement in average for 8 tasks (Table 2).\n\nWe hope this clarifies that our contributions extend beyond applying existing ideas.\n\n[1] Chen et al., Universal Self-Consistency for Large Language Model Generation\n\n---\n\n> W2. The paper proposes the first-thought prefix (FTP) to improve teacher quality but only validates it on a single model (Qwen2.5-32B-Instruct). It remains a question whether this specific prompting strategy is effective for other, more powerful reasoning models like DeepSeek-R1 or the OpenAI-o1, which may have different internal reasoning structures.\n\nAlthough we have not yet evaluated FTP on DeepSeek-R1 or OpenAI models because of computational-budget constraints, we agree that testing a reasoning model as teacher would strengthen our claim. We therefore experimented with a feasible open-source reasoning model, QwQ-32B. However, we observed that QwQ-32B solved most problems entirely inside its thinking steps without tools, so meaningful agentic traces are hard to be collected. This result suggests that strong reasoning models may bypass tool use, limiting straight-forward agent distillation. We have included this discussion to the revision and leave using reasoning models as the teacher for future work.\n\n---\n\n> W3. The experimental analysis, while broad, lacks depth in key areas. The individual contributions of FTP and SAG are not clearly disentangled, making it difficult to quantify their separate impacts on the final performance. Key hyperparameters, such as the sampling temperature for SAG, are presented without ablation studies to justify their chosen values.\n\nWe already provide an ablation of FTP alone and FTP+SAG in Table 2, showing that applying each method progressively improves performance. To further isolate the effect of SAG, we conducted additional experiments using SAG alone on the 0.5B, 1.5B, and 3B models. As shown in the table below, SAG alone improves average performance by up to +2.2 points (1.5B) and consistently yields benefits across all scales. \n\n|        | HotpotQA  | MATH500  | MuSiQue | Bamboogle | 2Wiki | GSM8K | AIME | OlymMAth | Avg    |\n|--------|-----|-----|----|----|----|----|----|----|--------|\n| D (3B)      | 48.4  |54 | 13  | 37.6 | 37.4 | 64.2 | 6.67 | 7.5  | 33.60   |\n| D+SAG (3B)  | 48.6 | 57.4 | 13 | 36 |37.4| 65.6 | 6.67 | 10  | **34.33**   |\n| D (1.5B)      | 43  |46.8 | 9  |27.2|35.6|54.8|1.1 | 7  |28.06   |\n| D+SAG (1.5B)  |43.8 |49.8 |11.6|31.2|36.6| 58 |7.8 | 3.5  | **30.29**   |\n| D (0.5B)    |34.6 |30.4 | 7  |17.6|28.8|31.2|3.3 | 1  |19.24   |\n| D+SAG (0.5B) | 34  |33.8 | 8.2|13.6| 33 | 33 |4.4 | 0  | **20.01**   |\n* D denotes the distilled agent.\n\nWe provide the ablation on sampling temperature in our response to Q1.\n\n---\n\n> Q1. In describing Self-Consistent Action Generation (SAG), you state that you sample with a \"high temperature\" to encourage diversity, yet the value used is 0.4. In many contexts, this is considered a relatively low temperature that promotes less randomness. Could you clarify this choice and provide any ablation studies you performed on the effect of temperature on SAG's performance?\n\nThank you for pointing this out. We acknowledge the term \"high temperature\" in Line 159 can be misleading. Our intent was to indicate a shift from deterministic decoding (temperature = 0) to sample diverse actions. We chose 0.4, as performance did not vary significantly across temperatures in SAG.\n\nAs suggested by the reviewer, we conducted a temperature ablation study on SAG for math tasks using the 1.5B model with temperatures 0.2, 0.4, 0.6, 0.8, and 1.0. The results are shown below and have been included in the revision.\n\n| Temperature | MATH500 | GSM-Hard | AIME | OlymMATH | Avg (Math) |\n|---|---|---|---|---|---|\n| 0.2         | 48.0    | 60.2     | **7.78** | 3.5       | 29.87       |\n| 0.4         | 50.6    | 60.6     | 6.67 | **4.5**       | 30.59   |\n| 0.6         | 50.8    | 61.8     | 4.44 | **4.5**       | 30.39       |\n| 0.8         | **52.4**    | 61.8     | 4.44 | 3.5       | 30.54       |\n| 1.0         | 51.0    | **63.8**     | 6.67 | 3.5       | **31.24**       |\n\n---\n\n> Q2. Could you provide more detail on the evaluation protocol? Specifically, were the reported scores, especially for benchmarks with small size like AIME, averaged over multiple runs with different random seeds to ensure statistical significance?\n\nWe report all experimental results with a single run due to computational budget constraints, as noted in Checklist item 7. We have clarified this detail in the experimental setup section of the revision.\n\nTo assess variance on smaller benchmarks, we additionally ran AIME with 5 different seeds for each model size with the FTP and SAG methods. The results below report the average and standard deviation:\n\n| Model       | Avg  | Std  |\n|-------------|------|------|\n| 0.5B  | 2.00 | 0.93 |\n| 1.5B  | 6.23 | 0.61 |\n| 3B    |14.44 | 1.36 |\n\nThese results show that the distilled agents yield **consistent performance on AIME** with relatively low standard deviation across model sizes. We have also included these statistics in the revised manuscript to enhance statistical rigor on the small benchmark.\n\n---\n\n> Q3. For the student models, you chose to use LoRA for fine-tuning. Given the relatively small size of both the models and the datasets, what was the rationale for using LoRA instead of full fine-tuning? Did you experiment with full fine-tuning, and if so, how did the results compare?\n\nThe primary rationale for using LoRA was its efficiency in terms of both reduced training cost and ease of deployment. Although we did not experiment with full fine-tuning, we believe this is a valuable direction and plan to include it in the future revision.\n\n---\n\n> Q4. An interesting trend appears on the MATH500 benchmark, where standard CoT prompting outperforms your distillation method for the 3B and 7B models, reversing the trend seen with the 0.5B and 1.5B models. Do you have a hypothesis for this scale-dependent performance difference?\n\nThank you for this insightful observation. Our hypothesis is as follows:\n\n- Smaller models (0.5B, 1.5B) benefit greatly from tool use, as it compensates for their limited arithmetic ability.\n- Larger models (3B, 7B) have stronger internal computation skills, making tool use less beneficial on simpler benchmarks like MATH500.\n\nHowever, **agent distillation remains effective for 3B and 7B models.** As shown in Figure 3, the 3B distilled agent performs better on harder MATH500 problems (level 5) with the agentic method.\nMoreover, on benchmarks involving more complex calculations (e.g., GSM-Hard, OlymMATH), the performance gains from agent distillation are consistent across 3B and 7B models. We have clarified this task-dependence in the experimental results section.\n\n---\n\n> Q5. The methodology states that you \"filter out any sequences that result in parsing or execution errors\" (lines 160-161). Logically, this filtering should result in a zero error rate for the final selected action. However, the results in Figure 6 show that SAG still produces a non-zero error rate. Could you clarify why the error rate is non-zero, or how is it calculated?\n\nWe thank the reviewer for highlighting this. The filtering rule, discarding samples with parsing or execution errors, **applies only when at least one valid action exists.**\n\nWhen all generated actions fail, we retain randomly-selected one failed action and feed its error message back as an observation, allowing the model to self-correct. The non-zero error rate in Figure 6 reflects these cases. We have clarified this in the method section to avoid confusion."}, {"review_id": "h2UQbO1JSN", "reviewer": "Reviewer_EVtf", "summary": "The paper introduces the Agent Distillation framework that distills tool-calling skills (e.g., search, code execution, etc.) from large-scale LLMs to smaller ones. The method proposes two key innovations: a new \"first-thought prefix\" (FTP) prompting scheme for enhancing the clarity of the teacher trajectories, along with the \"self-consistent action generation\" (SAG), which is a self-consistency method applied to the results of the code execution.", "questions": "See weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 1, "strengths_and_weaknesses": "Strengths:\n - The paper is written clearly and is easy to follow.\n - The paper presents evaluations on multiple benchmarks with extensive analysis.\n\nWeaknesses:\n - No particular novelty in the method -- same distillation loss, but the CoT trajectory is substituted with the agentic one. The prompting and self-consistency in generating code indeed enhance the quality of the trajectories, but have already been studied in the literature, e.g. [6].\n- [1] uses a large-scale LLM to collect a dataset of reasoning chains and tool calls and then applies DPO to train a smaller LLM. The proposed method should be compared to this baseline. Overall, the related works miss a lot of concurrent work, e.g., [4].\n - The main benefit over the original CoT distillation comes from using tools, not the distillation method itself, as can be seen by the higher average score of the teacher model when allowed to use tools, as well as a <= 3% difference in scores between distillation w and w/o FTP+SAG. Thus, it would be fair to compare with some of the tool-learning methods, like TinyAgent [2], ToolkenGPT, and others [3, 4].\n\n[1]. https://arxiv.org/abs/2410.18890\n[2]. https://aclanthology.org/2024.emnlp-demo.9.pdf\n[3]. https://arxiv.org/abs/2302.04761\n[4]. https://arxiv.org/abs/2502.05867\n[5]. https://arxiv.org/abs/2407.00121\n[6]. https://arxiv.org/abs/2309.17272", "was_revised": true, "final_justification": "The provided commentary on the difference between the existing work and underscoring the need for tools affected my decision to raise the score.", "rebuttal": "We thank the reviewer for the feedback and appreciate that they found our work (1) clearly written and (2) evaluated on multiple benchmarks with extensive analysis.\nWe believe some concerns may stem from a lack of clarity about certain aspects of our contributions. In the following, we provide explanation to clarify our novelty and distinguish our work from prior works.\n\n---\n\n> W1. No particular novelty in the method -- same distillation loss, but the CoT trajectory is substituted with the agentic one. The prompting and self-consistency in generating code indeed enhance the quality of the trajectories, but have already been studied in the literature, e.g. [6].\n\nThe contribution of our work is not based solely on methodological novelty. As acknowledged by other reviewers, our work provides well-motivated methods (NTqR, yrcR), comprehensive and extensive evaluations (qN16, NTqR, yrcR), and a valuable contribution to the field (qN16).\nWe also argue that **our method does introduce meaningful contributions beyond simply applying existing techniques**, as detailed below:\n\n- First, the First-Thought Prefix (FTP) is not a naive reuse of existing prompting methods. We identified a specific failure mode of LLM agents in reasoning tasks and proposed a novel solution: prepending an initial \"first thought\" derived from CoT prompting to better steer the agent's reasoning process. This approach results in improved performance for both the teacher (Table 4) and student models (Table 2). To our knowledge, **no prior work has applied similar approach to agentic reasoning or explored its advantages for distillation.**\n- Second, our Self-Consistent Action Generation (SAG) adapts the idea of self-consistency decoding to the agentic setting. Unlike prior methods that rely on ground-truth answers [1] or external LLM judges [2], our approach uses majority voting over sampled actions, based solely on observations from the environment. **This adaptation and its demonstrated effectiveness (Table 2) have not been explored in previous work, highlighting its contribution.**\n\n[1] Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models, ICLR 2023\n\n[2] Chen et al., Universal Self-Consistency for Large Language Model Generation\n\n---\n\n> W2. [1] uses a large-scale LLM to collect a dataset of reasoning chains and tool calls and then applies DPO to train a smaller LLM. The proposed method should be compared to this baseline. Overall, the related works miss a lot of concurrent work, e.g., [4].\n\nWe clarify that while our work and Manduzio et al. (2024) [1] share the high-level goal of training small models using trajectories from LLM agents, **their objectives, tool settings, training methodology, and evaluation scope differ substantially**, as summarized below.\n\n| Aspect           | **Our Work**                                      | **Manduzio et al. (2024) [1]**                                        |\n|---|---|---|\n| **Objective**    | Teach small models *interactive agent behavior* with tools            | Teach small models *function-calling ability* on fixed tasks     |\n| **Tool Use**     | Open-ended tools (retrieval, Python code) in math and factual reasoning          | Pre-defined functions (e.g., `add()`, `Stop()`) in simple logic/math tasks    |\n| **Training**     | Supervised learning on reason-act-observe trajectories                | DPO (preference-based learning from correct/incorrect completions) |\n| **Evaluation**   | 8 challenging reasoning-intensive tasks (HotpotQA, MATH, AIME, etc.)                         | 2 simple tasks (GSM8K, small FOL benchmark)                           |\n| **Target Small Model Sizes**   | 0.5B, 1.5B, 3B, and 7B    | 7B only         |\n\nDue to these differences, we believe a direct comparison would not be meaningful. We have cited this work and clarified its distinction in the revision.\n\n---\n\n> W3. The main benefit over the original CoT distillation comes from using tools, not the distillation method itself, as can be seen by the higher average score of the teacher model when allowed to use tools, as well as a <= 3% difference in scores between distillation w and w/o FTP+SAG. Thus, it would be fair to compare with some of the tool-learning methods, like TinyAgent [2], ToolkenGPT, and others [3, 4].\n\nThe teacher’s performance gain indeed stems from its ability to use tools. However, we argue that this fact **underscores the necessity and value of our agent distillation method.**\n\nOur central hypothesis is that **a student model cannot acquire tool-use capabilities through simple prompting alone**. As clearly shown in Table 2, directly prompting the student to use tools yields minimal or even negative gains, where 0.5B model achieves nearly zero accuracy.\n\nThis demonstrates that **only through distillation of the teacher's agent trajectories can tool-use ability be effectively transferred to the student.** In other words, agent distillation is the mechanism that unlocks the potential of tools for small language models (sLMs).\n\nFurthermore, while the 2-3 point absolute gain from FTP+SAG may appear modest, it is highly significant for sLMs, corresponding to **their relative gains of 13.83%, 8.87%, and 8.93% for 0.5B, 1.5B and 3B models, respectively.**\n\nTherefore, our work should be viewed not as a simple tool-learning method, but as a study that **identifies the limits of CoT distillation and pushes the capability boundaries of sLMs by the agent distillation paradigm.**"}, {"review_id": "9qo95HOJvO", "reviewer": "Reviewer_NTqR", "summary": "The paper introduces Agent Distillation, a framework for distilling agentic behavior into small LLMs. Besides naive distillation, the authors provide two precious insights that help boost results further. They train a family of models ranging from 0.5B to 3B parameters on 2 datasets, showing generalization capabilities on the other 6.", "questions": "- Could the authors provide additional experiments involving llama, phi, mistral, or other alternatives? It would bring more certainty about the Agent Distillation's strengths.\n- Have the authors thought of any ways of mitigating the downsides of the first-thought prefix (i.e., reduction of retrieval calls)?\n- Could the authors further discuss whether code execution during SAG is safe? Given the stochasticity of decoding, is there a risk of producing and executing dangerous or irreversible commands from the long tail of the model's distribution?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "## Strengths\n - Clear paper, full code: To me, the paper reads very clearly, and everything seems in place. I had a look at the code as well and it seems clear as well.\n - Well-motivated tricks with ablations: Both First-Thought Prefix and Self-Consistent Action Generation are simple but powerful ideas. The authors made sure to conduct enough experiments to show their strengths and potential downsides.\n - Cross-domain evaluation showcases generalization: The method is evaluated on eight diverse benchmarks covering both factual and mathematical reasoning, including out-of-distribution generalization. Small models consistently match larger CoT-based ones.\n - Small models and training set: Agent behavior is distilled into models as small as 0.5B parameters using only 1k-2k teacher trajectories.\n\n## Weaknesses\n - Evaluation on single model family: Even though it is acknowledged as a limitation, all experiments were conducted on Qwen models.\n - SAG poses some safety risks: Executing code sampled from a stochastic policy could lead to irreversible or unsafe operations.", "was_revised": true, "final_justification": "The presented method is clearly described and shows consistent improvements over baselines. I appreciate the additional evaluations shared during the discussion phase, which support the generality of the approach across different model families. The authors also included a complete codebase that seems clean and well-integrated with Hugging Face, which adds to the paper's practical value.\n\nThat said, I suggest the authors take into account the feedback from the other reviewers as well, especially regarding novelty and positioning relative to related work, even if I personally found the paper convincing. I recommend acceptance.", "rebuttal": "We sincerely thank the reviewer for the feedback and their appreciation of our work on multiple aspects.\nWe appreciate the reviewer's recognition of (1) our clear writing, (2) our complete code release, (3) thorough ablations, (4) simple yet effective techniques, and (5) strong generalization across diverse benchmarks. Below, we address each concern.\n\n---\n\n> W1. Evaluation on single model family: Even though it is acknowledged as a limitation, all experiments were conducted on Qwen models\n\n> Q1. Could the authors provide additional experiments involving llama, phi, mistral, or other alternatives? It would bring more certainty about the Agent Distillation's strengths.\n\nThank you for the thoughtful comment. We agree that evaluating our method across multiple model families would strengthen the generality of our findings. **We plan to extend our evaluation to other small models**, such as LLaMA-3.2-1B-Instruct and Phi-3-mini-instruct, in a future revision.\n\n---\n\n> W2. SAG poses some safety risks: Executing code sampled from a stochastic policy could lead to irreversible or unsafe operations.\n\n> Q3. Could the authors further discuss whether code execution during SAG is safe? Given the stochasticity of decoding, is there a risk of producing and executing dangerous or irreversible commands from the long tail of the model's distribution?\n\nWe appreciate the reviewer raising this important concern. In practice, **such risks can be mitigated through sandboxing mechanisms (e.g., Docker, E2B Sandbox)**, which are widely adopted in agent systems such as OpenHands [1]. Additionally, incorporating a safety guard model to verify code safety prior to execution is another viable future direction. We have clarified this in the discussion section.\n\n[1] Wang et al., OpenHands: An Open Platform for AI Software Developers as Generalist Agents, ICLR 2025\n\n---\n\n> Q2. Have the authors thought of any ways of mitigating the downsides of the first-thought prefix (i.e., reduction of retrieval calls)?\n\nOne possible solution is to **apply reinforcement learning after distillation** to encourage effective retrieval behaviors in students. While we did not pursue this direction in this work, we agree it is a promising avenue for future research in training small agents."}, {"review_id": "ZtLYIwK5Rs", "reviewer": "Reviewer_yrcR", "summary": "The authors introduce Agent Distillation, a training framework that teaches small language models to learn the reason-act-observe behavior of large tool-using LLM agents rather than just learning their chain-of-thought text. They introduce two techniques: a first-thought prefix that elicits teacher trajectories without extra fine-tuning, and self-consistent action generation that samples multiple tool-using trajectories at inference and chooses the one that yields a valid, consistent answer. Across eight factual and mathematical benchmarks, the distilled agents outperform standard CoT-distilled models.", "questions": "- One common pitfall of distillation is overfitting to teacher rollouts.  Did the author observe any overfitting in their experiments and how to prevent them?\n- How many rollouts did the authors collect from teacher per prompt?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n- The experiments are extensive and insightful, spanning across various model sizes (0.5B - 7B) and datasets (e.g., MATH, HotpotQA, AIME, Bamboogle). The authors also provide qualitative examples and failure case analysis.\n- The agent distillation idea is well-motivated and well-executed\n\n\nWeakness\n- The training method is a pretty straightforward extension of existing distillation methods and may slightly lack novelty. But a simple method that works well is a reasonable contribution.", "was_revised": true, "final_justification": "Issues resolved", "rebuttal": "We sincerely thank the reviewer for the thoughtful comments and positive assessment. We are glad that you found (1) the agent distillation idea well-motivated and well-executed, and (2) appreciated the breadth of our experiments, including the diverse benchmarks and model sizes. Below, we address the questions and concerns in detail.\n\n---\n\n> W1. The training method is a pretty straightforward extension of existing distillation methods and may slightly lack novelty. But a simple method that works well is a reasonable contribution.\n\nWe appreciate the recognition that a simple method with strong empirical results can be a valuable contribution. **Although our framework builds on distillation, it introduces two novel components tailored for agent distillation:**\n\n- First-Thought Prefix (FTP): Unlike standard prompting, FTP prepends a first thought generated via CoT prompting to guide the agent's reasoning, improving performance on reasoning-intensive tasks for both LLM (Table 4) and distilled agents (Table 2).\n- Self-Consistent Action Generation (SAG): his extends self-consistency to agent settings by sampling diverse action trajectories and selecting valid, executable ones based on environment feedback, thereby enhancing robustness and action quality.\n\nThese components, to our knowledge, have not been explored in the context of tool-using agent distillation.\n\n---\n\n> Q1. One common pitfall of distillation is overfitting to teacher rollouts. Did the author observe any overfitting in their experiments and how to prevent them?\n\nTo assess generalization, we evaluated the distilled agents on six out-of-distribution tasks, some of which are more challenging than the training tasks (e.g., AIME vs. MATH, MuSiQue (3-hop) vs HotpotQA (2-hop)). The consistent performance improvements over CoT-distilled baselines on those tasks suggest that **overfitting is limited**.\nWe also used LoRA-based fine-tuning, which is known to reduce overfitting compared with full fine-tuning [1].\n\n[1] Biderman et al., LoRA Learns Less and Forgets Less, TMLR 2024\n\n---\n\n> Q2. How many rollouts did the authors collect from teacher per prompt?\n\n**We collected one teacher rollout per prompt for training.** As suggested in prior work on CoT distillation [1], using multiple rollouts may further improve performance, which we plan to explore in future extensions.\n\n[1] Ho et al., Large Language Models Are Reasoning Teachers, ACL 2023"}]

Author rebuttal:

Meta-review: In this submission the authors introduce Agent Distillation, a new method for distilling model capabilities (such as reasoning and tool-use) from LLM agents into smaller models. Their approach utilizes LLM-generated trajectories of reason-act-observe cycles, and combines 2 approaches: 1) First-Thought Prefix, which enhances the quality of the teacher LLM's trajectories, and 2) Self-Consistent Action Generation, which improves the robustness of the student models at test time. The authors conduct empirical evaluation where they demonstrate the effectiveness of the proposed framework on factual and math reasoning tasks.

The reviewers agree that the idea behind agent distillation is clear and well-motivated, and that the paper itself is well-written and easy to follow. They also agree that the agent distillation offers a practical and effective pathway for transferring the agentic capabilities of LLMs into smaller, more efficient models, showing consistent improvements over baselines. The reviewers also appreciated the inclusion of the code with the submission, as well as cross-domain generalization results.

Overall, the reviewers unanimously agree that the paper provides clear and well-justified methodology, backed by rigorous evaluation. I am therefore pleased to recommend this paper for acceptance.
Decision: Accept (spotlight)