Unlocking SLM Potential for Data Analysis Code Generation via Non-Parametric Knowledge Distillation
Authors: Jinyang Li
Jack Williams
Nick McKenna
Arian Askari
Nicholas Wilson
Reynold Cheng
Venue: NeurIPS 2025 poster ()
Keywords: LLM
Knowledge Distillation
Data Code Generation
Primary area: applications

Abstract: Knowledge distillation from Large Language Models (LLMs) to locally hosted Small Language Models (SLMs) provides advantages for Data Analysis Code Generation (DACG) such as privacy protection. However, achieving effective distillation without resource-intensive training is challenging. This paper investigates whether LLMs can distill knowledge to SLMs through In-Context Learning (ICL), a training-free method for rapid task adaptation. We present the DarGO: Distillation and Adaptive Reasoning-Guided Orchestration framework, which facilitates automatic knowledge distillation from LLMs to SLMs. DarGO consists of three phases: exploration through an Model Orchestration Interface (MOI), Memory Collection of successful trajectories, and Knoweldge-driven Inference. We evaluate DarGO on three challenging DACG benchmarks (WikiTQ, TabMWP, and Bird-SQL), each with in-domain training sets that enable detailed analysis of knowledge distillation effectiveness. DarGO demonstrates a substantial relative performance improvement of 27.5\% on average for the student SLMs. To further observe generalization capabilities, we evaluate the \method across different teacher-student model combinations, knowledge transfer scenarios, and unified memory approaches for more advanced, test-only data analysis tasks. Our findings contribute a novel perspective on distillation methods that enhance high performance for SLMs while avoiding intensive fine-tuning.

Unlocking SLM Potential for Data Analysis Code Generation via Non-Parametric Knowledge Distillation
Jinyang Li :aack Williams", Nick McKenna", Arian Askari", Nicholas Wilson", Reynold Cheng" "Microsoft Research Cambridge The University of Hong Kong j107258cconecct hk, jack. aaiimmmmmmieee crosoff. com, ckcheng@cs hku.hk
Abstract
Knowledge distillation from Large Language Models (LLMs) to locally hosted Small Language Models (SLMs) provides advantages for Data Analysis Code Generation (DACG) such as privacy protection. However, achieving effective distillation without -ssurcceinteesive training is challenging. This paper investi- gates whether LLMs can distill knowledge to SLMs through In-Context Learning (ICL), training-free method for rapid task adaptation. We present the DARGO: Distillation and Adaptive Reasoning-Guud Guidedd Orchestration framework, which facilitates automatic knowledge distillation from LLMs to SLMs. DARGO con- sists of three phases: exploration through an Model Orchestration Interface (MOI), Memory Collection of successful trajectories, and Knoweldge-driven In- ference. We evaluate DARGO on three challenging DACG benchmarks (WIKITQ, TAEMWP, and BIRD-SQL). each with in-domain training sets that enable de- tailed analysis of knowledge distillation effectiveness. DARGO demonstrates a substantial relative performance improvement of 27.5% on average for the student SI.Ms. To further observe generalization capabilities, we evaluate the DARGO across different teacher-student model combinations, knowledge transfer scenarios, and unified memory approaches for more advanced, test-only data analysis tasks. Our findings contribute a novel perspective on distillation methods that enhance performance for SLMs while avoiding intensive fine-tuning, The source code is available: https: : /gtttbb.. com/
Introduction
Data Analysis Code Generation (DACG) automates the conversion of natural language queries into executable code, empowering information extraction and analysis from tabular data efficiently. This process enhances productivity, reduces the technical barrier for data analysis, and allows data scientists to focus on deriving insights, ultimately supporting more effective decision-making [24, 17, 14]. This a challenging task since not only requires the capability of code generation but also understanding complex tabular data.
Large Language Models (LLMs) have demonstrated remarkable performance across diverse, complex tasks [49, 37, 8, 71, 11]. Leveraging LLMs or LLM agents for automatic code generation from user queries offers effective solution [65, 56]. However, the integration of LLMs in DACG faces two primary challenges: 1) Privacy concerns arise when utilizing closed-source LLMSs such as GPT-4 [2] Claude-3.5-Somme [40]. 2) Deploying large open-source models like Llama-3. 1.m5--311400BB 1-405B [13] or DeepSeek-v3 (671B) [31] can be challenging due to their large number parameters. Balancing these benefits and challenges is crucial for effective data science applications.
Work done during Research Intern at Microsoft Research Cambridge
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Agoni Ondeevndionittttee
Hazner Sotttration
6g Knowledise-- drhenn Inference
Dorect Tripe Bey SOR
JIAD bet Metuleittttee Biutiaakee
Cervec Trijector with COULRCCI
Woone Swwesary Ad wwd.eaa
d d'dendelgg in Mording Cimeerttetttii
Cac State
O
Frne
Figure 1: Overview of the orchestration system of DARGO for data science code generation. Left: Model Orchestration Interface (MOI) with abstraction lifting, orchestrated coding, and plan opti- mization. Center: Memory Database Construction, including trajectory collection and case study integration. Right: Knowledge-drive Inference and Planning. featuring RAG-based meta instruction generation, and knowledge-drine -riiven code generation. R: Referenced GT Code. F: Functional Plans. O: Orchestrated Coding.
Small Language Models (SLMs), such as Phi-3-mini [1] and Llama-3. [13], have gained attention for their In-Context Learning (ICL) capabilities and ability to be locally deployed. These models offer computational efficiency and enhanced data privacy, crucial for resource-constoe constrrindd or privacy- sensitive applications [21]. While SLMSs have shown competitive performance in some general tasks including natural language understanding [39] and code completion [7]. their effectiveness in data science code generation tasks remains an open question.
Fine-tuning is a common strategy to enhance SLM capabilities for complex tasks [42]. However, this approach encounters several challenges in the domain of data science DACG. One primary issue is the limited availability of high- -ush-iualityy training data. Professional tabular datasets, such as relational databases, are often small or proprictary, restricting access to substantial corpora for training. Additionally, the dual expertise required in both coding syntax and data understanding for accurate annotation further constrains dataset scalability [30, 28]. This is reflected in recent benchmarks for data science code generation, which typically contain around or fewer than 1,000 samples, highlighting the complexity and resource constraints in this field [20, 3, 26, 70, 66]. Recent research has explored distillation from LLMs to SLMs through fine-tuning on synthetic data [52, 36, 22]. While this approach shows promise, several challenges persist. For example, the performance improvements obtained from fine-tuning approaches can fail to generalize across different programming languages or dialects, requiring re-training for each package update or new task [46, 23 However, In-Context Learning (ICL) can adapt new requirements or tasks by providing relevant instructions examples, reducing the effort required for re-traiming or continual training. This raises our central research question in DS code generation: Can LLMs distill knowledge to SLMs through In-Context Learning (ICL)?
In this paper, we explore the potential of knowledge distillation from LLMs to SLMs via ICL. We propose a novel Distillation and Adaptive Reasoning-Grus Guineed Orchestration framework that enables an LLM to serve a Teacher model guiding SLMs (Student models) in complex DACG tasks. DARGO operates in three phases: exploration, memory database collection, and knowledge- driven inference. During exploration, we employ the Model Orchestration Interface (MOI) that allows an LLM to probe and analyze SLM code knowledge by converting questions into step-wise functional plans and asking SLMs to infill the code for each plan. Then, successful collaborated cases are stored in a memory databases. In the knowledge-drive inference phase, DARGO utilizes a Aatrieval-augmentt
generation (RAG) approach that dynamically distills knowledge at inference time by presenting relevant prior successful cases in a case-study format to guide the generation process, which proves friendly to SLMs to absorb (Section 2.3).
We evaluate DARGO on three challenging tabular analysis benchmarks requring code genera- tion:TABMPP [34), BIRD-SQL. [29], and WIKITQ [41]. where each feature fixed, non-overlapping train-test splits to closer show how effective of knowledge distillation that DARGo is. The exper- imental results demonstrate that the DARGO framework significantly improves the performance

---

Dotabase Inputt
oucdica Poxt
Ordhestiaeed Codes
Arte bai RewuAd)
Den MA
number
Hidket susigginn
Question:
>>
Sep Court dooour eemarry fiiee MUET (1BR1)
Which
Yiu conperpie 1eo
Cataltion CiNCOS/tieee
Sep ir de ammeett
orageny Roonapory Conagor
Calo prososaage Accemendayy Aite SCLECT pelcammeen aocamress __iine 100 3/ Jirnn
61 IOA 330 COMN177 finud righ
TI ielegrr 4412 cokpprr..
inctal from
Figure Main steps of MOI demonstrated with a text-to-SQI. task example. Teacher model converts referenced code to a functional plan for Student models to complete. Teacher model iteratively optimizes the plan until the Student model produces correct code. A Python code MOI example is provided in Appendix D.I.
of SLMs across all datasets, validating the potential of our knowledge distillation approach via ICL. Importantly, the memory produced for one student can guide other students, demonstrating model-agnostic transfer. Additional cross-dataset experiments on CRT-QA [70], QRDATA [32], and Infi-Agent [20] confirm that the distilled knowledge generalizes beyond the original training distribu- tion. Taken together, our results show that lightweight, arivcc-preserving SLMs can inherit much of an LLM's analytic expertise through ICL alone, making DACG practical in resource-comstrained or privacy-sensitive environments.
2 Methodology
2.1
Task Formulation
Given a natural language query 4s E Q, where Q = (q1 92.....99. 2Nr represents a set of N queries, the corresponding tabular data database schema information d; D, where D {dl, d2 ... dy }. then the Small Language Model (SLM) is tasked with generating an executable code snippet c, This code snippet must accurately answer the query di with the associated data d4. The function that maps each query-data pair to its corresponding code snippet by SLMs is defined as sgen and can be written
C Sgm (:..g)) for i = 1,2,....
2.2 Model Orchestration Interface
The Model Orchestration Interface (MOI) is conducted between a Large Language Model (LLM), Teacher model, and a Student model, represented by the SLM. A Teacher LLM with superior reasoning capabilities will often generate plans that are too abstract for weaker Student SLM to enact. These plans require decomposition and refinement match the operational granularity of the Student SLM. Through orchestrated mediation, the MOI dynamically adjusts plan granularity to align with the SLM's code capabilities whilst maintaining task conformance. The MOI composed of three key components: Abstraction Lifting, Orchestrated Coding, and Plan Optimization.
Abstraction Lifting (AL). In this phase, LLM generates a functional plan P; = (811, 8:2..... 8:k) based on query q. data input de and the corresponding ground truth (gt) code &. The ground truth code & can either be sourced from an existing dataset (BIRD-SQL) or generated by the Teacher model when not directly available but ground-trutt answer string exists, such as WIKITQ and TABMWP. This functional plan is defined as (a 4, &), where La denotes the LLM performing abstraction lifting. Each step Siy in the plan corresponds to a key subtask derived from the query, collectively forming a structured template outlining the solution process. These steps are annotated by the LLM with descriptive comments and placeholders such as [Fill Your Code Here] in Python or [Fill Your Sub-Query: in SQL,, shown in Figure 2, ensuring that the SLM follows the logical flow of the entire plan and enables guided code generation. Unlike Chain-Of. Thought [60] plans, which provide intermediate steps in continuous textual form, our approach
3

---

Table 1: The 6 action types utilized by the LLM during the Model Orchestration Interface (MOI) to optimize the plans for better understanding and code generation by SLMs
Action Type
Expression
Description
Decomposition
step (x) + step (a),
Split a complex step into smaller, manage- able steps such as step a and step b.
step (b)
ALT
step (x) step (y)
Replace step described by ambiguous incorrect messages with clearer and correct alternative step
ADD
step (x) + step (x),
Add a necessary stop to ensure the com- pleteness code logic.
step (a)
DELETE
step (x) + None
Remove the unnecessary step which may lead to misunderstanding by the SLM.
SIMPLIFY
step (x) +
Replace complex step with simpler approach. For example, convert recursive plans into iterative loops.
[mpll_steppx)
SWITCH
packagea. step (x) +
Use simpler package to achieve the same functionality. For example, conversion from Package Lirear Regress on to Correlation Coefficient to deter- mine relationship between two variables.
packageB.step (x)
bridges high-level problem understanding with low-level code implementation logic, allowing the SLM to more effectively follow the plan for DACG.
Orchestrated Coding (OC). Once the functional plan P: provided, the Student SLM consid- ers all context including the question and data input to generate the complete orchestrated code c; = Syen(d, qs. P.) in a single turn by filling all placeholders, ensuring the solution is correct and executable. The results from executing this orchestrated code are then compared to those from reference solution (such as ground truth answer string or gt codes) evaluate whether the SLM fully understands both the data and the logic needed to answer the question. This comparison serves as a key indicator of the problem-solving accuracy of SLM and alignment with the intended solution. While the ground truth code may already be available from datasets or generated by the Teacher model, orchestrated coding and abstraction lifting are crucial for a few reasons. First, AL breaks down complex problem-solving tasks into manageable sub-tasks, with the potential to improve the performance the SLMs across a wide range of analytical queries by assisting them in understanding modular structure. Additionally, error isolation can be grounded in the program structure, enabling more precise identification issues and contributing to optimized plans. This is supported by our analysis in Section 3.6 that compares chain-of- of-thought with functional plans for exploration.
Plan Optimization (PO). The plan optimization process an iterative procedure that unfolds over multiple turns, denoted by t. During each iteration, the SLM refines the functional plan P. To formalize this interactive optimization process, we define an environment & = (S, A, O, T) following [72, 62, 16]. where S represents the state space, A the action space (Table 1), and O the observation space. this context, the plan P is embedded within the current state S serving as structure that guides the SLM to generate code. The orchestrated code of is the snippet produced by performing the plan Pt within the environment.
During each turn, the LLM observes of, the outcome of executing orchestrated code of generated by the SLM, and selects an action af from A to optimize the plan. For example, if a step si contains an error such as: "Step j: List players who was born before 1930 and after 1 1950", the LLM will apply an action ALT (+) to correct this, resulting in an updated plan P with the refined step s/+1 as "Step j: List players who were born after 1930 and be bero 1950" This iterative process can be represented as:
p++1 = {s/1 $
(+l
(2)
.+1 Capt (s.g of at). 1g

---

Here, si s+1 is updated by the optimization function Copt of the LLM, which integrates the current observation of, action of and sub-optimal step s/g. The system transitions from state S S through A resulting in the updated plan p The SLM then generates the updated orchestrated code c [genlgs, d, ) for the new plan. This process repeats until the output is correct the maximum number of iterations T is reached.
2.3 Memory Database Construction
After interactions between the LLM and SLM in MOI, the finalized states are stored in a memory database. This database includes the correct orchestrated codes, along with the context of the question and data input. This process ensures that the SLM can efficiently reference and apply related knowledge new, unseen queries.
eey Case Study Average Weight Calculatoo Speditc Players
What the average wiight of Janant Sandon and Roburt Will lama?
Take
"Clurine" Height Weight Faskion, Elorattwnr, Pres bus Teansj)
Janon Sanden: Weight 210 be
eee Ottlecive:
Explenation
Miaadd Dasy": cssa lnsde tob-oepacted crluec
"paow Rowx wow..ooooooodda compponding niwarr Stndee" Robbert Williaee fhtood oow dibout
""Durpart The aeu peleted innege
hefhilwing these
underttod how
minuitiine Albewiidts Thicrase
devoncrntts prottio opelictiioo of esto pulpubt andinna laa ddng soodic Pytton.
Figure An example of generated case study to enhance comprehension for SLMs.
Case Study Translation. Rather than only storing raw, heterogeneous cases that consist of query, plan, and orchestrated code in simple stacked format, the LLM refines these into case study-like representations. These representations distill the reasoning behind the success of each example, serv- ing as intermediate abstraction that emphasizes the underlying rationale for the chosen approach. Each case study G; contains a Case Name, Question Schema / Value Informat ion, Ob jeet ive, and an Explanat ion of how the solution code successfully addresses the query using the provided data. An example of this structure is provided in Figure 3. As shown in Figure (a)-(b), DARGO performs case study translation only for correct cases, because reflecting on incorrect cases without supervision often introduces hallucinations.
Correct Case Collection. The Correct Case Collection, denoted as M, consists of cases where the SLM generated correct orchestrated codes. Each case Mj this collection contains the natural language query q., the corresponding data d, the correct orchestrated code é, which contains descriptive comments as shown in Figure (right), and the case study G, illustrating the solution. The set M is the union of all such individual cases:
M M M-UM. Mc,
M, = (qx, d4,,d,, Ccs Gt).
2.4 Knowledge Distillation from Memory Database
This part presents a RAG-based method of knowledge distillation as a global instruction, termed as meta-instruction. This instruction then guides the SLM in learning how to plan and generate code more accurately for unseen queries.
RAG-based Knowledge Distillation for Meta Instruction Generation. We propose a Retrieval- Augmented Generation (RAG) framework for localized instruction distillation. In the retrieval phase, we identify the top relevant examples from memory database M via embedding model. Relevance measured via function DD expressed as R(g:) = D(g,, M, k), where is number of most relevant cases.
5

---

Model
WIKITQ
TABMWP
BIRD-SQL
Accuracy
Grad. 146
Grad. 7-8
Total
Sim.
Med.
Chal.
Total
CodeLLlama-715
11.80
26.55
13.11
20.50
43.92
18.00
11.76
24.40
CodeLlama-1338
34,90
37.27
24.22
45.27
19.60
17.65
26.80
StarCoder2-7B
20.70
34.00
27.56
31.10
41
21 31.60
17.65
26.60
StarCoder211B
36.60
39.09
33.14
36.50
43.92
29.20
14.71
30.60
Phi-1-Small-7B
27.00
46.36
38.00
42.60
52.03
28.40
10,78
31.80
SLM Performance
Phi-3-Minnl3.880
32,50
44.18
38.89
41.80
38.51
21 21.20
11.76
24,40
Chain- Chain-Or:Thoone Thouught
27.70
46.36
35.33
41.40
34,46
22.00
12.75
23,80
Static Fow-Shor
23,00
37.27
34.89
36,20
47.97
20.80
7.84
26,20
Dynamic Few-Shon
16,60
51.45
52.89
52.10
42.57
18.90
11.76
24.40
DSPy Distilatiinn Shot)
ReGAL Distillation
36.10
44.73
38.
41 80
35.14
14.80
12.75
20.40
Instruction (Ouns)
51.35
Table 2: Performance comparison of various SLMs on WIKITQ, TABMWP, and BIRD-SQL, with
results presented in accuracy percentages. Improvements of our DARGo methods over the End-to- End Code Gen baseline are highlighted using different intensities of olive color. Bold indicates best results for Phi-3-Mini, while underlines denote second-best results. means hybrid with dynamic few-shot with DarGO. Detailed instructions of when to use few shot is in Appendix A.4.
Then, shown in Figure 1 (d). the case studies of these retrieved examples are then fed into the SLM to extrapolate plans for solutions, adhering them to the specific query. Here, SLM performs a sec- ondary distillation, extracting shared knowledge patterns from these case studies, which have already been distilled by the LLM (Teacher model), to generate instructions, noted as Meta-Instruche Instructiin (Im (4:)), precisely specific to the current query at hand. The process is formalized:
[m(())) = fage Uis R(a))
where fagg is aggregation function applied by the SLM. By doing so, SLMs generate more relevant and contextually appropriate instructions, effectively bridging the gap between general knowledge and query-specific requirements.
Knovledge-Driven Inference. Harnessing the distilled instructions I = Im(q;) from the memory database, the SLM initially formulates structured plan Pgen= which it subsequently employs to generate code for new queries (Figure (e)). For a given query q and its associated data d,, this process unfolds follows:
Pgen fplan di, q)), = fga((d, Ri, Pga)),
(5)
where Splun denotes the planning function executed by the SLM. This plan serves as blueprint, guiding the following code generation phase. The SLM then employs the function fgens which takes Pgen along with the original query qi and data de generate the final code G.
Experiments
In this section, we first describe datasets and evaluation metrics in Section 3.1, followed by imple- mentation details Section 3.2. We then present a comprehensive experiments aimed at addressing three key research questions:
RQ1 (Section 3.3): How effective is DARGO for DACG?
RQ2 (Section 3.4): Does DARGO and the knowledge it distills generalize across models?
RQ3 (Section 3.5): How does DARGO compare popular Lora fine-tuning?
RQ4 (Section 3.6): Are all components of DARGo necessary?
3.1 Datasets and Metrics
We evaluate our approach on the DACG datasets WIKITQ [41], TABMWP [34), and BIRD-SQL [29]. These datasets vary data complexity and task requirements. Full dataset statistics are provided in Appendix B.I. WIKITQ features operations such counting, comparison, and aggregation, and

---

Table 3: Zero-shot performance of DARGO on different SLMs. Improvemenss (in parentheses) show gains over the Baseline.
Table 4: Zero-shot performance of DARGO on different teacher LLMs. + means distill knowl- edge from teacher LLMs to student SLMs.
Model
WIKITQ TABMWP
BIRD-SQI.
WiKITQ
TARMWP BIRD-SQL.
StarCoder-153 Student
Llama-3.3-708
Phi-3-mini
Baseline
36.60
36,50
30.60
Baseline
41.80
32.50
24.40
DARGO-MM
45.70 c9.10)
43.93 (17,))
38.40 (7780)
DARGO-MI
48.80 (7700)
42.60 ent..0)
36.20
Llama-3. 3.1-88 Student
Llama-3.3-700
Llama-3.1-88
Baseline
34.30
42,90
40.20
Baseline
42,90
34.30
40.20
DARGO-MM (15.90) (76
(04
DARGO-MI
50.40 (7.50)
42,00 (17.70
47,00 (15,800
select 1,000 test and 2,000 training examples. Accuracy is measured using the official scripts from Pasupat and Liang [41]. TABMWP involves mathematical word problems in tabular data, with 1,000 questions used for memory construction and a separate 1,000 test set. Performance is evaluated by comparing answers to ground truth. BIRD-SQL presents relational databases with both semantic parsing and analytical tasks. We adopt the 1.000-example mini-train set from Qu et al. [44] and evaluate 500-example mini-dev set using execution accuracy (EX).
3.2 Implementations
Setup. We conduct experiments in two settings. For TABMWP and WIKITQ, we follow Stengel- Eskin et al. [50] to first have Teacher LLM (GPT-4o) generate silver programs during exploration. At inference, an SLM produces Python code, which is executed to obtain a final answer string for comparison with the ground truth answer. For BIRD-SQL, silver code is required because the dataset already contains ground-trut truth SQL. RAG-based Meta Instructions employ h = 3 nearest neighbors (via CodeT5+ [59]). Further implementation details appear in Appendix B.
Baselines Models. We consider an SLM suitable it can perform in-context learning and has under 15B parameters for one A 100 80G GPU inference. For closed-source models, choose GPT-35-Turbo as SI.M since it has faster inference speed and its performance falls behind advanced models (e.g., GPT-4). We classify them into three categories: Orchestration Models, where GPT-44o the Teacher, and Phi-3-mini-128k [1], Llama-3. 1-8B [13] serve as Students for verifying the generalization of our approach. Evaluation Models, including Phi-3, CodeLlama, and StarCoder2 families. Knowledge-Tre Snoowldge-Tranmmission Models, containing GPT-35-Turbe and Llama-3.1-80) used to test knowledge transmission Section 3.4.
Baseline Methods Implementation. Baseline methods include zero-shot end-to-end, Chain-of- Thought [60]. static few-shot [6], and dynamic RAG-basee few-shot [15] with memory database by DARGO since python tasks do not have GT codes but answer string, each with three examples. We also compare two distillation frameworks, DSPy [25] and ReGAL [51], both using GPT-4o as the teacher.
3.3 Overall Results
Table 2 highlights three key aspects:
1) Effectiveness of DARGo: Distilling knowledge via DARGO propels Phi-3-mini to outperform both End-to-End Code Generation and Chain-of-Thought all SLM datasets, achieving relative gains of 17.5% on TABMWP to 38.5% on BIRD-SQL. Moreover, Phi-3-mini with DARGO often surpasses larger models (2-3x parameters), outperforming 1odeLlama-13B by 7.0% and StarCoder2-195 by 3.2% on BIRD-SQL, SQL, and rivaling Phi-3-Medium (4x parameters) across all benchmarks.
2) Comparison with Advanced Distillation Strategies. We implement two advanced distillation pipelines, DSPY and REGAL, on GPT-4o and Phi-3-mini for fair comparison, reporting the best performance of REGAL across different numbers of reusable helper functions (1-10) to minimize bias. As shown in Table 2, DARGO outperforms both approaches on all three datasets and difficulty levels.
7

---

Performance Knowledge Transmission
70
End2End
COT
57.7
DarGO-MI (ours)
Performance (%)
50.6
50.4
40.7
46.6
42.942
45.2
40.239.6
44.4
41.6
34.3
20
WIkiTQ
WikiTQ GPI-35-Turbo
TABMWP GPT-35-lurbe
BIRD-SOL Llama-3.1 3ama-3.1-88
BIRD-SOL GPT-35-lurbo
Llama-3.11 3.1-88
3.1--3.1-88
Figure 4: Knowledge transmission for from the memory database between GPT-4o and Phi-3-mini across three datasets.
DSPY is robust for general QA but struggles in execution- DACG settings because its exact- match metric does account for valid alternative program solutions. Moreover, DSPY relies few-shot demonstrations rather than comprehensive knowledge distillation, reflecting the broader challenges of demonstration-baat task-specific instructions to guide complex operations, shifting focus from prompt optimization to experience summarization.
methods for complex DACG. By contrast, DARGO prioritizes
We also evaluate REGAL, which provides Reusable Python Helper Functions from the memory database. Although promising for tightly scoped, domain-specifie tasks, REGAL proves less ef- fective for cross-domain DACG (see Table 2). Error analysis shows it often produces narrowly tailored functions (e.g., get_ _least_pointstt _eam_m Filter_senator ________errrr________ which often directly reference the data schema and are therefore irrelevant on new tasks. In contrast, the distilled knowledge of DARGO is textual, focusing on reasoning- challenges, and is therefore not syntactically fixed to specific domain or input.
3.4 Generalization
Here, we conduct experiments to test generalization of DARGO framework:
Is DARGO model-agnostic to SLMs? To investigate the breadth of the DARGO pipeline beyond single teacher-student configuration, we perform two complementary experiments. (1) We apply the complete workflow to two additional student models: StarCoder2-152 and Llama-3. 3. 1- while keeping GPT-4c as the teacher. As summarized in Table 3, both students outperform their respective baselines all three datasets, indicating that DARGO consistently improves variety of SLM architectures. (2) To examine robustness with respect to the teacher LLM, we replace CPT-4o with Llama-3 3. 30B. The results in Table 4 replicate the earlier gains, demonstrating that gains of DARGO are not confined to a specific teacher model.
Is the distilled knowledge only useful the student model participating in Orchestration? Next, we examine whether the knowledge distilled through DARGo is only beneficial to the student model involved in the orchestration phase. Even though this has been proven effective, it would be costly if exploration phase is repeated for new SLMs. Therefore, test the distilled instructions two new SLMs, L1 ama -3. 1--1- and GPT 35 3orrbo,, both of which have not participated the exploration between GPT 4o and Phi - m-mmi.i
Figure 4 shows that meta instructions (MI) derived from our memory database (jointly updated by GPT-4o and Phi- Phi-3-mini) produce siginificant improvements over baselines. Specifically, Llama-3. 1-8B achieves an average relative improvement 14.3%, while 30.9% gain. These results indicate that the distilled knowledge is not tied to specific student model; can effectively transfer to new models without additional fine-tuning, offering scalable means of augmenting emerging SLMs.
sees

---

Table 5: Performance of DARGO on different SLMs. Improvements (in parentheses) are over the Baseline.
Table 6: Ablation study of exploration in DARGO BIRD-SQL..
Model
CRTQA
QRDATA
INFI-AGENT
METHOD
BIRD-SQL (MT)
Phi-I-mint as Student
DARGo w/ Phi-3-mini
33,80
Baselinc
2651
31.14
42,80 (73
DARGO-MI
32 12000
e7a9)
(a) who MOI
21.70 (112.10)
Homm--3 1-88 as Student
(b) w/o Case Study Trans.
22.80 (1.1100)
Baseling
34,43
(c) w/o Functional Plan
23.60 (210.220)
+ DARGO-MI
45.88 (1104)) 44.81 1510
54.18 (1
Evaluation of DARGO Current tabular- tabular-reasoning benchmarks frequently provide test-only splits due large costs of more complex tasks annotation, prompting us to evaluate how well DARGO generalizes when no in-domain training data are available. Following the [55], we first construct unified memory from our exploration corpus and evaluate its performance on those test-only sets via DARGO. To be specific, exploration set B contains: WikiTQ [41], TabMWP [34], BIRD-Pandas [29], and Juice [3]. The distributions of B are shown in Appendix B.4. The test-only
datasets form the set E, which includes CRT-QA [70], QR-DATA [32], and Infi-Agent [20].
During evaluation, when we test DARGO on benchmark & E &, its corresponding exploration corpus is B U & This strict partition guarantees zero overlap between examples used for distillation and those reserved for evaluation. across all student models, demonstrating that it distills transferrable reasoning patterns rather than memorizing speesst- artifacts.
summarized in Table 5, DARGO presents huge OOD improvements
3.5 Fine-Tuning v.s. DARGO Knowledge Distillation
With the Same Seed Data. We compare DARGO with LoRA fine-tuning [19] using the same 1,000 training samples on BIRD-SQL for Phi-3-mini. Figure and Table in Appendix shows that fine-tuning limited data even degrades SLM performance, while DARGo achieves significant improvements. We attribute this to two factors: I) small training sets introduce bias that limits generalization; and 2) LoRA struggles to teach complex reasoning capabilities in end-to-end regime. In contrast, DARGO leverages LLMs to decompose difficult questions into interpretable steps and distill planning knowledge, enabling better generalization. This makes DARGO particularly effective for DACG domains with limited annotations.
Comparison with Incremental Lora Fine-Tuning. For more comprehensive comparison, we conduct additional experiments fine- tuning Phi-3-mini on varying proportions (10% 20%, 50%, 70%, 100%) of the full BIRD-SQL training set. Each configuration is repeated times, with average results presented in Figure 5. Our findings reveal several notable patterns: First, performance peaks at 70% of the data, suggesting that fine-tuning on more data does not always lead to better results, which maybe due to the inclusion of lower-quality or biased examples in the full dataset. Second, while fine-tuning with 70- 10-100% of the data marginally outperforms DARGO, it requires 7-9x more training examples to achieve this performance. Notably, DARGO outperforms fine-tuning on 50% of the data, demonstrating 5 x greater data efficiency.
122
Purfomaace Ratio
0.a
20%
70%
100%
Percentage Training
Comparison of DarGO with Lora fine-tuning across varying data size.
Figure 5:
3.6 Ablation Study
In Table 6, to better validate the function of each component of DARGo, ablation studies performed in both Exploration and Inference (More details in Appendix E):
(a) MOL. For the "w/o MOI" scenario, we construct the memory by employing GPT-4o to convert ground-truth code to plans directly, excluding the participation of SLM. This resulted in significant final performance drop for Phi-3-mini. One explanation is that GPT-4o tends to produce relatively coarse-grained plans (c.g., "compute percentage of winning") "), whereas SLMs require

---

more fine theeggained steps (e.g., "compute total number of games", "divide winning games by total wins"). These higher-level plans can induce hallucinations to SLMs, as they frequently entail additional intermediate steps that increase the risk of errors. Furthermore, without execution-bail -ased refinement and calibration through the Plan Optimization (PO) component in MOI, plans generated by GPT-40 often contain inaccuracies that can mislead the SLM, as illustrated in Section 2.2.
(a) Case Study Translation. As elaborated in Section 2.3, case study translation systematically consolidates and refines heterogeneous inputs, including tabular data, user queries, and code samples, into a structured, context-rich representation that SLMs can more effectively process. In the absence of this translation component, the model frequently defaults to errorous or incomplete outputs (e.g., "SELECT \n In Wnnnn/nn.. .n)....), underscoring the difficulty SLMs encounter in performing generalized reasoning when confronted with raw, unprocessed data in DACG.
(c) Functional Plan. A key contribution of our work is the MOI and the use of functional plans to structure and improve the interaction between teacher and student. We evaluate performance by replacing functional plans with textual plans like Chain-of-Thoughh The result can prove that our designed functional plans are better orchestration media type compared to general COT plans in data science code generation task.
4 Related Work
Data Analysis Code Generation (DACG). DACG automates code generation for data-centric tasks in formats such as CSV, TSV, and relational databases (RDB) [7, 35]. requires code that accurately handles schemas, formats, and data semanties, whether in Python for tabular data [9, 10, 47] SQL code generation further extends DACG to formula generation in tools [54, 5]. Although large language models have shown promise, privacy remains challenge in cloud-based environments [35].
for databases [68, 27, 29]. Spreadsheet-baxet
Knowledge Distillation. Knowledge distillation can mitigate this problem by transferring LLM capabilities to smaller models, enabling efficient deployment in resource-constrained environments [63]. The field has evolved from early work on softened output training [18] to advanced techniques like task-specific fine-tuning [45], zero-shot learning [57], and instruction-followinn datasets [58, 57]. Progressive distillation techniques, such the Orca framework [38], demonstrate the potential for guiding the development of efficient open-source models. Self-distillation approaches have explored autonomous training data generation [58]. Recent advancements have focused on improving the performance and privacy aspects of DACG by knowledge distillation [35]. At the same time, synthetic data has been leveraged to enhance the generalization of SQL generation across different
schemas [64]. Even though these techniques are effective, most still require training efforts to transfer knowledge. Our DARGO framework introduces distillation through in-context learning, eliminating the need for task-specific fine-tuning.
Memory for Large Language Models. Memory can enhance LLMSs by retaining long-term context and knowledge [69], as in reflection- based frameworks [48], subroutine reuse [50], and self-correction [4]. Complex tasks may require repository-level memory [65, 56, 55J. Typically, memory remains confined to a single model. Our work introduces multi-model memory orchestration (via GPT-4o), enabling smaller models to tap into broader knowledge sources for DACG.
5 Conclusion
In this paper, we presented DARGO, an automatic framework for knowledge distillation from Large Language Models to Small Language Models in Data Science Code Generation. DARGO leverages In-Context Learning to enhance SLM performance without fine-tuning, using model orchestration and memory-based distillation to improve task accuracy. Evaluations on three challenging tabular data analysis datasets that require code generation show a 27.5% relative performance increase for Phi-3-mini. We also show model-agnostic effectiveness, benefiting other SLMS, even they did not participate in the orchestration. These results highlight the potential of DARGO for developing intelligent applications with a focus on privacy and computational efficiency.
10

---

Acknowledgments
We thank the anonymous reviewers and committees for their helpful comments, suggestions and organizations. We appreciate Xinnuo Xu, Christian Poelitz, Siān Lindley, Hank Lee, Xiaolong Li, Ge Qu for discussion and suggestions. Reynold Cheng, Jinyang Li, were supported by the Research Grant Council of Hong Kong (RGC Project HKU 17202325), the University of Hong Kong (Project 2409100399), and the HKU Faculty Exchange Award 2024 (Faculty of Engineering).
References
[1] Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, al. Phi-3 technical re- port: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08574, 2023.
[3] Rajas Agashe, Srini Iyer, and Luke Zettlemoyer. Juice: large scale distantly supervised dataset for open domain context-based code generation. F.NNIP-JJCNIP, pages 5436-5446, 2019. URL httph::t///c http:///ccaaattotcccoeo
org/Di 9-1546.
[4] Arian Askari, Christian Poelitz, and Xinye Tang. Magic: Generating self-correction guideline for in-context text-to-sql. arXiv prepriní arXiv:2406. 22622, 2024.
[5] Kushal Bhatia, Sumit Gulwani, Raksha Saikia, and Samuel Samuel. Shectcopilot: Assistive table formula completion with neural code generation models. In Proceedings of the 36th Anmual ACM Symposium on User Interface Software and Technology, pages 701-714, 2023.
[6] Tom B. Brown, Benjamin Mann, Nick Ryder, and Others. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[7] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107 100.:2007..3374. 2021.
[8] Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Trans. Mach. Learn. Res., 2023, 2023.
[9] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. In The Twelfth International Conference on Learning Representations, 2024.
[10] Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong. Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. Binding language models in symbolic languages. The Eleventh International Conference on Learning Representations, 2023.
[11] Xiang Deng. Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards generalist agent for the web. In Thirly-seventh Conference on Neural Information Processing Systems, 2023. URL https: http:::///ppeeevvi..o net/ Forum?id=ke dorumf?id=kkyyp003ww..
[12] Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Rühle, Laks V. S. Lakshmanan, and Ahmed Hassan Awadallah. Hybrid LLM: Cost-efficient and quality- aware query routing. The Twelfth International Conference on Learning Representations, 2024.

---

[13] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[14] Usama M. Fayyad, Gregory Fiatetsy-SShapit and Padhraic Smyth. From data mining to knowledge discovery in databases. Al Mag., 17(3):37-54, 1996.
[15] Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. Text-to-sql empowered by large language models: A benchmark evaluation. Proc. VLDB Endow, (7(5)) 145,, 2024.
[16] Yu Gu, Yiheng Shu, Hao Yu, Xiao Liu, Yuxiao Dong, Jie Tang, Jayanth Srinivasa, Hugo Latapie, and Yu Su. Middleware for Ilms: Tools are instrumental for language agents in complex environments. arXiv preprint arXiv:2402, 14672, 2024.
[17] Jiawei Han, Micheline Kamber, and Jian Pei. Data Mining: Concepts and Techniques, 3rd edition. Morgan Kaufmann, 2011. ISBN 978-0123814791.
[18] Geoffrey Hinton. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.022 020.00331, 2015.
[19] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[20] Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma. Guoyin Wang. Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, Yao Cheng, Jianbo Yuan, Jiwei Li, Kun Kuang, Yang Yang. Hongxia Yang. and Fei Wu. dnfiagent-dabench: Evaluating agents on data analysis tasks. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21 21-27, 2024, 2024.
[21] Harshit Joshi, Abishai Ebenezer, José Cambronero Sanchez, Sumit Gulwani, Aditya Kanade, Vu Le, Ivan Radiček, and Gust Verbruggen. Flame: A small language model for spreadsheet formulas. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 22995-13033 2024.
[22] Minki Kang. Seanie Lee, Jinhcon Back, Kenji Kawaguchi, and Sung Ju Hwang. Knowledge- augmented reasoning distillation for small language models in knowledge-intensive tasks. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[23] Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pre-training of language models. In The Eleventh International Conference on Learning Representations, 2023.
[24] Mohammad Khanbabaei, Farzad Movahedi Sobhani, Mahmood Alborzi, and Reza Radfar. Developing an integrated framework for using data mining techniques and ontology concepts for process improvement. J. Syst. Softw. 137:78-95, 2018
[25] Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan A, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. DSPy Compiling declarative language model calls into state-of -fte---fffhhhaeee -hrt pipelines. In The Twelfth International Conference on Learning Representations, 2024. URL https: http//pppneeeeeeeeeeeeeeee net / forum? i id= = YNOO YoOd.
[26] Yuhang Lai, Chengxi Li, Yiming Wang. Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen- tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pages 18319-18345. PMLR, 2023.
[27] Chia-Hsuan Lee, Oleksandr Polozov, and Matthew Richardson. Kaggledbqa: Realistic evalua- tion of text-to-sql parsers. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing, ACI/IJCNLP 2021, (Volume 1: Long Pupers), Virtual Event, August 1-6, 2021, pages 2261-2273. Association for Computational Linguistics, 2021.
12

---

[28] Fangyu Lei. Qian Liu, Yiming Huang, Shizhu He, Jun Zhao, and Kang Liu. S3Eval: A synthetic, scalable, systematic evaluation suite for large language model. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1259-1286, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
[29] Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. Can Ilm already serve as database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36, 2024.
[30] Jinyang Li, Nan Huo, Yan Gao, Jiayi Shi, Yingxiu Zhao, Ge Qu, Yurong Wu, Chenhao Ma, Jian-Guang Lou, and Reynold Cheng. Tapilot-crossing: Benchmarking and evolving Ilms towards interactive data analysis agents. arXiv preprint arXiv:2403.05300 2024.
[31] Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chenggi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v22 v2: A strong, economical, and efficient ofxture-of-experts language model. arXiv preprint ar arXiv:2405.0444
2024.
[32] Xiao Liu, Zirui Wu, Xueqing Wu, Pan Lu, Kai-Wei Chang, and Yansong Feng. Are LLMs capable of data-based statistical and causal reasoning? benchmarking advanced quantitative reasoning with data. In Findings of the Association for Computational Linguistics ACI. 2024, August 2024.
[33] Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Noua- mane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19175 2024.
[34] Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chur Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In International Conference on Learning Representations (ICLR), 2023.
[35] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardeoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations, 2024.
[36] Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models to reason. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1773-1781, Toronto, Canada, July 2023. Association for Computational Linguistics.
[37] Yao Mu, Qinglong Zhang. Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. Embodiedgpt: Vision-language pre-training via embodied chain of thought. Advances in Neural Information Processing Systems, 36, 2024.
[38] Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.00 02707, 2023.
[39] Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July
5-10, 2020, pages 4885-4901. Association for Computational Linguistics, 2020.
[40] Taiwo Blessing Ogunseyi, Cossi Blaise Avoussoukpo, and Yigiang Jiang. A systematic review of privacy techniques in recommendation systems. International Journal of Information Security, 226)):65511166, 2023.
[41] Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACI. 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers, pages 1470-1480. The Association for Computer Linguistics, 2015.
13

---

[42] Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thore. Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2523-2544, Online, June 2021. Association for Computational Linguistics.
[43] Mohammadreza Pourreza and Davood Rafiei. Din-sql: Decomposed in-context learning of text-to-sql with self-correction. Advances in Neural Information Processing Systems, 36, 2024.
[44] Ge Qu, Jinyang Li, Bowen Li, Bowen Qin, Nan Huo, Chenhao Ma, and Reynold Cheng. Before generation, align it! a novel and effective strategy for mitigating hallucinations in text-to-sql tol generation. arXiv preprint arXiv:2405. 15307, 2024.
[45] Sanh. Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter. arXiv preprint 11X1:11100001008, 2019.
[46] Da Shen, Xinyun Chen, Chenguang Wang, Koushik Sen, and Dawn Song. Benchmarking lan- guage models for code syntax understanding. In Findings of the Association for Computational Linguistics: EMNLP 2022. pages 3071-3099 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[47] Li Shen, Yicong Xu, and Hongbo Wang. Handling ambiguity in natural language code genera- tion. Transactions of the Association for Computational Linguistics, 10:1 13-126, 2022. doi: 10.1 111162/tal_aa00444
[48] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[49] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 1 11523-11 11--15535 1530. IEEE, 2023.
[50] Elias Stengel-Eskin, Archiki Prasad, and Mohit Bansal. Regal: Refactoring programs to discover generalizable abstractions. In Forty-first International Conference on Machine Learning. ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
[51] Elias Stengel-Eskin, Archiki Prasad, and Mohit Bansal. Regal: Refactoring programs to discover generalizable abstractions. In Forty-firs International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024.
[52] Gemma Team, Morgane Riviere. Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprins vrXiv:2488.00111 2024.
[53] Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. RAT- SQL: Relation-aware schema encoding and linking for text-to-SQL parsers, In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, Online, July 2020. Association for Computational Linguistics.
[54] Daming Wang, Yijun Liu, and Qiang Zhang. Spreadsheet formula generation using lightweight language models. In Proceedings of the 37th AAAI Conference on Artificial Intelligence, pages 2023.
[55] Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better I1m agents. In ICML, 2024.
[56] Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Opendevin: An open platform for software developers generalist agents. arXiv preprint arXiv:2407.16741 2024.
14

---

[57] Yizhong Wang. Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Processing Systems, 36.74764-74786, 2023.
[58] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct: Aligning language models with self-generated in- structions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 13484-1355 13508. Association for Computational Linguistics, 2023.
[59] Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. Codet5+: Open code large language models for code understanding and generation. arXiv preprint htt::23050079222 2023.
[60] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting clicits reasoning in large language models. Advances in neural information processing systems, 14:24824-24837. 2022.
[61] Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian- Yun Nie. C- pack: Packed resources for general chinese embeddings. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '24, page 641 1419, New York, NY, USA, 2024.
[62] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint 2024.
[63] Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey knowledge distillation of large language models. arXiv preprint arXiv:2402. 13116, 2024.
[64] Jiaxi Yang, Binyuan Hui, Min Yang, Jian Yang. Junyang Lin, and Chang Zhou. Synthesizing text-to-SQL data from weak and strong LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, August 2024.
[65] John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering, arXiv preprint 2xXv::2405.15733 2024.
[66] Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Oleksandr Polozov, and Charles Sutton. Natural language code generation in interactive data science notebooks. In Proceed- ings of the st Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), Toronto, Canada, July 2023.
[67] Junchi Yu, Ran He, and Zhitao Ying. THOUGHT PROPAGATION: AN ANALOGICAL APPROACH TO COMPLEX REASONING WITH LARGE LANGUAGE MODELS. In The Twelfth International Conference on Learning Representations, 2024. URL https:
//operevieew net/ / Sorum?id=SBoRheee
[68] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Tan, Murhaf Fawzi Er, Iris Li, Jun Ma, Zilin Li, and Dragomir R. Radev. Spider: A large-scale human-labeled dataset for complex and cross- doss-- semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911 1-1211, 2018. doi: 10.18653/v1/DI 1-1111111.....1
[69] Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. A survey on the memory mechanism of large language model based agents. arXiv preprint ar arXin:2404.13501, 2024,
15

---

[70] Zhehao Zhang, Xitao Li, Yan Gao, and Jian- Guang Lou. Cit-qa: A dataset of complex reasoning question answering over tabular data. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2131-2153, 2023.
[71] Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) is a generalist web agent, grounded. In Forty-first International Conference on Machine Learning, 2024. URL https: /ppprreiie....eee.....m.m net forum? wett//ooommmdddieeeeee ddm.imiedppiiii11
[72] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307. 13854, 2023.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim that are the first work to discuss how well training- free knowledge distillation can achieve.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitation and future work in Appendix G
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We show instructions of reproducibility in Appendix K
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification:
We provide the codebase in supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nig) cc/
public/ gug CodeSu'om ss Lonolicy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/nnpss. cc/ cc/publicee suublic/ge //////uaaa i des /CodeoodSuboo ss onool Pooolice icy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provide implementation details Appendix B.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For each inference result, results, which can prove that the performance show statistical significance.
experiments 5 times and present average
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We stated that all SLM inference based on one NVIDIA A100 80GB GPU card and how much cost for distillation in Appendix I.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /tte::////uuippsp http:///ccccccccccccce.eeeeeeeee
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Detailed Appendix J. Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cited them.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers' aw. thcode com daaaaaaaas sete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https: : / /////neerppppp//pp//ee cc/Confeeeecccs / 2025/
LLM) for what should or should not be described.
23

---

A
Model Implementation
We implement models for three main categories of purpose:
A.1 Orchestration Models
gpt-4o: The Teacher model (gpt-4o) is responsible for several key tasks, including Abstraction Lifting (see Section 2.2) and Plan Optimization (see Section 2.2), which are performed while monitoring the performance of the Student model. Additionally, the Teacher model handles the conversion of complex, heterogeneous cases into more readable case studies for Student Learning Models (SLMs), as detailed in Section 2.3.
Llama-3. 3 generalization of DARGO workflow.
Ien--Instruuut ruct: we use this powerful open-source model as teacher to prove the
phi-3-mi 22k--122kiimm nst ruct: For the orchestration process, we select this 3.8B parameter SLMM as the Student model due to its strong generalization abilities and efficient deployment.
1lama 1 11..1-8----- instruct
A.2 Baseline Models
Within the orchestration mode, several families of Student Learning Models (SLMs) are evaluated. These include models from the Phi-3, Starcoder 2, and LIDARGo families:
Phi-3 Family [1]
mai-33-mnn--12--insstte (3.8B)
mal-3--saal-122k-inntttz (7B)
p (14B)
Starcoder 2 Family [33]
sta starcoder2-7--insttee
starcoder2-155-instrre
Llama Family [13]
codellama- instruut-hh
A.3 Models in Knowledge Transmission
In Section 3.4, we explore the knowledge distilled from DARGO to newly developed models, particularly in terms of their ability to generalize knowledge. For this evaluation, we select the following models:
1lama-3. 1- 1--b--intttrcee This model is broad new, yet it shows significant performance improvements when leveraging the distilled knowledge.
gpt mpt--5-cuzbo-166t We also include a source model our experiments to demonstrate the effectiveness of approach aeross both GPU-deployed and API-based models. Despite its number of parameters is unknown, we consider as one of SL.Ms since its performance falls behind of its more advanced versions such GPT-4.
A.4 Demonstration V.S. Distillation
Given the memory database, we compare the effectiveness of our knowledge distillation techniques, with conventional demonstration- based strategies. In our approach, distillation involves transferring knowledge from the memory database to SLMs through task-specific instructions. On the other hand,

---

demonstration-base based methods guide SLMs by presenting explicit task examples to facilitate analog reasoning [67]. We implemented two variants of few-shot demonstrations: Static: Human experts select three representative examples from the memory database, which remain constant across all cases. Dynamic RAG-based: Examples are selected from memory database based on similarity to the current query. For fair comparison, we also implement the same RAG system as DarGO-MI, described in Section 3.2.
Our findings show that few-shot demonstration generally underperforms DarGO across datasets. However, RAG-based few-shot demonstration outperforms both our designed knowledge distillation and other baselines on TABMWP. This success appears linked to the simplicity of TABMWP's input data, which averages 2.22 columns and 6.13 rows per data point, with clean numeric or processed string values (Table 7). In contrast, for WIKITQ with irregular value types, and BIRD-SQL with complex schemas, SLMs struggle, generating 38.2% more invalid outputs, such as erroneous SQL queries, in BIRD-SQL.. Based on these observations, we conclude that dynamic few-shot demonstration is more convenient and effective for leveraging the memory database when the input data is less complex. On the contrary, for complex data such as tables with dirty values or relational databases, our designed knowledge distillation enables SLMs to better utilize knowledge and perform tasks more effectively.
B Dataset Implementation Details
B.1 Train-Tess Distillation Data Statistics
Table 7 summarizes the distributions the datasets used in this study. These datasets in- clude both single-table and multi-table relational structures, offering a comprehensive aspects for evaluation. The question types range from standard semantic parsing queries to more complex analytical questions involving aggregations and nested operations. This diversity enables thorough assessment of the model's performance across various query paradigms.
Table 7: Statistics for three datasets. The term Analysi is indicates that the dataset mainly con- sists of analytical questions, while SP refers to semantic parsing tasks.
B.2 Data File Content
For convenient reproduction and following, we preprocess all dataset into more unified data format of jscnl. In python task (TABMWP, WIKITQ), each line of data contains question_is id,, question, data_path, answer_type,
STATISTIC
TARMWP WIKITQ
BIRD-SQL
Dataset Features
train examples
L,00
L,000
eval examples
1,000
1,000
500
data_ overview, answer. In SQL task (BIRD-SQL), each line of data contains quest on_ on_id, question, evi dence, data_path, db_ id, sql.
question type
Analysis 26.5
SP 12.6
SP Analysis
each
A toks Q
20.0
Data Structure
data input type
Single 6.13
Single 28.5
RDB
A rows / data
354k
columns data
2.22
73.3
B.3
Data Input Content
Code Features
code type
Python String
Python String
SQL.
The main goal of this work is to evaluate the code generation capabilities of models in un- derstanding data schemas and structures across multiple datasets. Given the impracticality of providing data values in real-world scenarios in which datasets may consist of millions or even billions of rows, sample values for the part of data input to simulate realistic code generation tasks. We feed the markdown format of schemas with data samples as tata_overview.
answer type
Code
toks
N/A
N/A
61.15
For TABMWP, we provide only the column names and the first three rows of values. This enables models to infer the data structure and value types necessary for Python Pandas code generation without exposing all the data.
For WIKITQ, which contains more complex and varied value types, we provide the first 10 rows of values and column names help models navigate the dataset's intricacies.
In the case of BIRD-SQL, which contains relational databases with complex schemas and diverse value types, more advaneed schkma-linking techniques are often required to retrieve relevant tables
25

---

or columns before answering queries [53, 43]. While we consider this advanced schema binking-inkkng process as future work for DARGO, our current focus on the code generation aspect. Therefore, we provide:
Ground truth retrieved tables, reducing input complexity and simulating realistic human- machine interactions where users might supply potentially relevant tables.
Full columns with column meaning description files.
The first three rows of values for each table.
Although the retrieved tables are given, the models must still consider constraints and generate correct SQL queries. As shown in Table 2, performance on Bird-SQL remains relatively low, even with simplified table retrieval, highlighting the challenges of generating accurate SQL queries in complex database environments. This methodology allows us to evaluate code generation capabilities while approximating the real-world challenges of data analysis.
B.4 OOD Cross-Dataset Evaluation Set Statistics.
Table show the data statistics of basic set of unified memory for evaluating DARGO OOD cross-dataset distillation.
Table 8: Statistics for Basic Set of Unified Memory.
Dataset # Items
wikitq
000
TabMwp
000
BIRD-Pandas
300
Juice 100
C Action Types
Decomposition. The Decomposition action type divides a large, multifaceted step x into multiple simpler steps and b. By breaking complex workflows into smaller components, the Model Orchestration Interface (MOI) ensures that the resulting plan is both clearly understood and more straightforward for SLM to execute. This approach clarifies the logic behind each sub-step and allows finer-grained control over how tasks are performed or combined, thereby reducing confusion and facilitating future refinements.
ALT. Sometimes, the instructions for step can ambiguous, incorrect, or unnecessarily complex. The ALT action type replaces such problematic step with newly clarified step y. By substituting erroncous or unclear instructions, the MOI ensures that the plan adheres to more accurate logic, minimizing the likelihood of misinterpretation and promoting consistency throughout the workflow.
When critical operation is missing or an additional step is required for completeness, the ADD action type introduces a new step into the existing plan. Adding steps proves valuable when the plan overlooks essential checks, transformations, or other auxiliary procedures. This mechanism ensures more thorough and dependable solutions.
ADD,
Certain steps be redundant or risk causing confusion for subsequent code generation. The DELETE action type removes any unnecessary step x, thereby streamlining the plan. By eliminating irrelevant instructions, the MOI reduces cognitive load on the SLM and maintains logically consistent sequence of steps that aligns directly with the overarching goal.
DELETE.
SIMPLIFY. Whenever possible, it can be advantageous to simplify complex steps. The SIMPLIFY action type replaces a complicated step x with more direct version, simple__sted (x). For example, it may transform solution relying on recursion into an iterative, loop-based approach. Simplification improves both computational efficiency and interpretability, since SL.Ms often perform better with concise instructions.
26

---

Table 9: Performance evaluation of Zero-Shot End-to-Eie Code Generation, LoRA tinettuning, and our proposed knowledge distillation techniques on BIRD-SQL. Deeper red shading indicates a larger performance drop compared to the original rre-trained model, while green indicates no decline or improvement.
Model
SIMPLE
MEDIUM
CHALLENGING
OVERALL
Zero-Shot End-to-End Code Gen.
Original Checkpoint
38.51
21.20
11.76
24.40
LoRA Fine-Tuned
39.86
19.20
10.78
23.60
DARGO Knowledge Distillation
Meta Instruction
51.35
30.40
16.67
33.80
SWITCH. Selecting appropriate library or tool is critical for achieving both correctness and simplicity. The SWITCH action type replaces a step implemented with packageA by an equív- alent (or more suitable) operation from packageB. For instance, one might switch from using LinearRegressil to the relationship between two variables. This approach avoids unnecessary overhead and preserves clarity in the plan.
aatiinoooffiiiie when the task is simply to determine
By employing these six action types, Decompos i t on, ALT, ADD, DELETE, SIMPLIFY, and SWI TCH, the MOI systematically refines high-level plans. This process results efficient and easily interpretable workflows, ensuring consistency from the design of the plan to its final implementation by SLMs.
D DARGo Functionality
D.1 MOI Generalization
GT Codes
Functional Plan
Orchestrated Codes
Import pandas pd
Load the data
Import numay as np of pd.read, covi..
2
[Fil Your Code]
pd.aa eswiflle, def)
Clein the data for acts
cleon the data for acts. and stages
4 dfflacts') di acts lstr extractt ...
4
[FII Your Code]
4 for col in columas:
dff stagesl off extract
difcol) pd.to_numeett numerief.))
offlacts
(di
stages
di stages (soone) (d0)..)) / Atd0
[Fil Your Code]
threshold
9 outtiers df absiddf acts zecore
9. dil ascoree Idft acts stal]
compute
10
10 Compute spores
10 drj (ttges, 25core)) (dif -1 stdl))
Answer the Question
Your
11
outliers.empyy:
Answer Question
print)"Yes")
Question
print)Yes"") outiiers ("No')
elses
14 [FI Your Code)
15
print""No")
15
Figure 6: Illustration of how MOI conducted Python for Tabular data analysis.
Our Model Orchestration Interface (MOI) is adaptable to different programming languages with different data input settings. Figure shows how MOI is conducted in RDB settings with SQLite, and Figure 6 shows how it's undertaken in Single-tabular data with Python.
D.2
Fine-Tuning v.s. DARGO Knowledge Distillation
E
Ablation Study for Inference
We conducted a comprehensive ablation study of DARGO-MI, shown in Table 10. Code-T5+ is code embedding model [59], while BGE-Large [61] represents one of the sfate-of-the-arn (SOTA) text embedding models. The study examines two types of RAG Index: one where distance is computed using question embeddings alone, and another where both question and schema embeddings are used. 'Pan Gen" approach involves first constructing a plan with distilled knowledge, followed by generation using knowledge-drive planning. In contrast, the "Gen" approach involves direct
27

---

generation without prior planning, The instruction type labeled w/ examples refers to cases where specific example is provided by the Teacher model. We evaluate performance with 3. and examples to assess the impact of varying numbers of RAG examples. The results of the ablation study reveal several key insights:
Code embeddings outperform text embeddings. The superior performance of Code-T5+ over BGE-Large-en can be attributed the nature of the task. While text embeddings emphasize on semantic and domain knowledge, code embeddings capture the syntactic and logical structure of coding problems, which is crucial for DACG tasks. Even when presented with identical questions, the code solutions can vary significantly depending on the data input. Code-T5+ is able to effectively embed questions from a programming perspective, benefiting from its pre-trained corpus, whereas text embeddings are less suited for the task.
Embedding only the question more effective than embedding both the question and schema. The study demonstrates that question-on onnly embeddings lead to better results. This suggests that the inclusion of schema in the embedding may introduce unnecessary complexity, which may hinder performance on the DACG task.
Planning essential for more complex tasks. The results stress on the importance of planning in prowledge-driven generation. For tasks requiring complex reasoning, the "Plan + Gen" approach outperforms direct generation (Gen), indicating that structured planning significantly improves task performance.
One example may bias the SLM. Involving single example in the instruction can introduce bias in Small Lanuguage Models (SLMs). A specific example might cause the SLM to over-follow to certain information, leading to hallucinations. For instance, if the example includes a reference to " singer" ", the SLM may generate plans that include "sinner" even when the question pertains to an unrelated topic, such as "cars". This observation highlights the lack of robustness in SLMs when exposed to overly specific examples. Consequently, it is better to provide more general, transferable knowledge instructions. The degraded performance observed with 1 RAG example supports this conclusion, as the model becomes overly reliant on the provided information.
More examples do not always improve performance. Interestingly, increasing the number of RAG examples (from I to 5) results in performance drop. This suggests that longer input sequences may confuse the SLM, making it more difficult to distill relevant knowledge. Based on these findings, we recommend using 3 RAG examples as the optimal balance for complex DACG tasks since it avoids both the biases of a single example and the confusion caused by too many examples.
Table 10: Ablation Study Results of DARGO-MI of Phi-3-mini on BIRD-SQL. The table compares different embedding models, RAG index (with without schema), reasoning approaches (planning or direct generation), and varying numbers of RAG examples.
Embbeddin Model
RAG Index
Reasoning Type
Instruction Type
RAG Examples
Performance
code-15+
question
plan + gen
no examples examples
33.80
code-15+
question
31.40 (12:40)
hge-large
question
plan + gen
no examples
30.00 (13 50)
code-15+
question
plan gen
w/ examples
28.00 (15.50)
code-15+
question=schemm
plan + gen
no examples
32.40 (31.40)
code-15+
question
plan gen
examples
31.80 (1200)
code- 15+
question
plan + gen
no examples
29.80 (11.00)
F Error Analysis
We conducted an error analysis by sampling 50 incorrect cases for both DARGO-MI across three datasets. Although DARGO substantially improves the overall performance of SLMs, we found that 54% of the errors were caused by over-reasoning. This issue tends to emerge even relatively simple cases. As discussed earlier, SLMs can overly adhere to the instructions derived from planning and guidance, which is problematic when the task enough simple and does not require decomposition or reasoning. In these cases, direct code generation would lead to more accurate results. The remaining
28

---

WHNTO
Figure 7: The scaling performance of DARGO on three main DACG datasets.
emrors stem from common issues in code generation tasks, such as incorrect string handling, incorrect column selection, database constrain understanding.
G Limitations and Future Work
A key limitation of our current approach with DARGO is the reliance on initial training examples for both LLMs and SLMs to facilitate orchestration. This is why we selected datasets that include a training corpus suitable for distilling knowledge. However, important avenue for future work is to explore how to generate such training data in a fully zero-shot manner, without relying on human- annotated or enumerated examples. Additionally, as highlighted in the error analysis, over-reasoning negatively impacts performance on simpler tasks, where additional reasoning or decomposition is unnecessary. To address this, future work could focus developing prompting smaller models to act as routers, as proposed by Ding et al. [12], to classify questions based on whether they require planning. This would help avoid over-reasoning in straightforward cases and improve the overall efficiency of DARGO.
H Scaling Analysis of DARGO
Figure shows that DARGO scales reliably with increasing data volume. Although its performance is somewhat unstable on very small memory, stability improves rapidly, and accuracy surpasses the baseline once more than 60% of the exploration data are available which proves that the distilled knowledge is being exploited effectively.
I
Cost Analysis
We compare the cost of DARGO with other distillation methods shown in Table 12, Table 16 and Table 14. And Table 11, Table 15, Table 13 show the inference cost of DARGO and its comparison with other baselines. We can conclude that: first, it shows DarGO now is the most efficient Distillation work for this task. Second, since inference is conducted by SLMs, the additional token usage is acceptable when considering its significant improvement.
Table 11: BIRD-SQL: Infer (Tab. 16)
Method
SLM InToks
SLM OutToks
EX 1
End2End (baseline)
358,277
16,751
24.40
Ohain-Of-Thought
369,433
19,317
23.80
Static Few-Shot
794,342
17,306
26.20
Dynamic Few-Shot
836,678
26,291
24.40
DSPy
637,146
60,748
22.80
ReGAL
1,602,698
16,045
20.40
DarGO-MI
647,799
64,836
33.80
J Broader Impact
Given DARGO is designed for automating code generation for better and efficient data analysis, can help data scientists explore the potential risks of financial market, carthquake, which are beneficial to

---

Table 12: BIRD-SQL: Exploration (Tab. 17)
Method
LLM In InToks
LLM OutTokk +
Cost 4
DSPy
18, 18,129,304
2,689,076
$108.32
ReGAL
13,259,759
2,381,368
S85.44
DarGO
11,086,194
1,372,421
S62. 16
Table 13: WikiTQ: Infer (Tab. 18)
Method
SLM InToks |
SLM OutToks +
Acc.
End2End (baseline)
235,122
44,033
32.50
242,443
149,246
27.70
Static Few-Shot
521,292
45,161
23.00
Dynamic Few-Shot
549,075
93,415
16.60
DSPy
418,131
123,422
26.70
ReGAL
2,082,898
162,949
36.10
DarGO-MI
495,493
168,311
41.10
the society. Also, our study only focuses on code generation, a programming-level language rather than natural language, will not impact society negatively.
K
Reproducibility
We provide codebase in the supplementary files and we list all implementation details in Appendix B And we deliver prompts for each stage and baselines in Appendix for fully reproducibility.
L
Main Prompts
The zero-shot End-to-Ent tnd Code Generation prompt is shown in Figure 8, Figure 16 and 18 show the zero-shot Chain- Chain-OF-Thhughh reasoning. Figure 19 shows few-shot demonstration prompting. The few_shot__ cerrt__eeepp_eeee ls be selected by human experts Statie Few-Shot Demonstra- tion, and can be retrieved from DARGO memory database by RAG system as Dynamic Few-Shot Demonstration.
The Figure 9, 10, show prompts for Orchestration between LLMs and SLMs. Figure 12 presents how LLM convert orchestrated successful cases to more understandable case studies to SLMs. LLMs can go through correct cases from memory databases and distill knowledge to an offline and plug-and and-plan General Instruction for SLMs to used for new and unseen queries performed by prompts shown in Figure 13 and 14. During inference, SLMs can produce Meta Instructions by prompts in Figure 15. Given distilled knowledge (instructions), SLMs will plan first as shown in Figure 17, and generate codes finally with their knowledge-driven planning, which shows in Figure 18.
M Knowledge Distillation Examples
M.1 Case Study Example
The Figure 20 shows the example case studies on Python task. Figure 22 present examples of DARGO-MI respectively.
30

---

Table 14: WikiTQ:
Exploration (Tab. 19)
Method
LLM InToks
LLM OutToks
Cost
DSPy ReGAL
10,893,180 7,967,176
1,616,141
$65.09
1,431,207
$51.34
DarGO
6,661,181
824,828
$37.35
Table 15: TabMWP:
Infer (Tab. 20)
Method
SLM InToks
SLM OutToks
Acc. Score
End2End (baseline)
171,480
38,016
41.80
Chain-Of-Thowe Thoughtt
176,764
39,244
41.40
Static Few-Shot
380,177
38,283
36.20
Dynamic Few-Shot
400,358
52,578
52.10
DSPy
304,898
49,073
40.40
ReGAL
767, 114
37,679
41.80
DarGO-MI
314,817
125,230
45.70
Table 16: TabMWP: Exploration (Tab. 21)
Method
LLM InToks +
LLM OutToks +
Cost +
DSPy
9,426.638
893,604
$48.75
ReGAL
7,790,084
751,736
$40.49
DarGO
5.330,507
659,862
$29.89
analyet,
date, you generate
question:
Please Follow:
Do not add data inspection in the plann, such as 'df- or "print Idf, head())" since this is
Do act any externel intomettion.
The code should id be end-to-end, so you cannot encourage yourself to print ocher things except
finel rexult. More other infortation lead ta be distracted.
Question: [question)
Shought: need to xee the date zarplex the first rowes
Code:
"pyython
impoxt pandas pc
of re.resd, csvi' (ooii""sst lidsts_pathi', sepe"ltt) pelae idr, hhedd(
Observation:
Thought:
question:
'python
import pandas cs
Figure 8: Prompt of baseline end-to-end generation for tasks requiring Python.
31

---

generate trainer, educating you oonwers your P You enalysia questiions. let your students to Elll underdtand plans and analysis. So you camnas generate code by your own, only generate glana.
path (dete_path) (Cirst
Question: iquestion)
Original
"pythor
(ground truth code)
You should convect code into coda_pian fozmaz placahalder "*ccodo_ee _ode_plan
TPILL youR HERE]'s
import
Step
[F:11 Your Codc]
Step
[P:l]
Step M:
Ginerate your codc_ptann plan for your studert. Do BOT generate any code by your own. Aise ignora and ranove stepe or inepersing the deta which leads to scudent cheating. Please note iste hard for student write long code. will 1, 1,000 dollare if have good
Figure 9: Prompt converting ground-truth code to functional plan for python task example. This is conducted by LLM Teacher model.
data, expert functionsl plan, complete each line of code to anawer questions correctly:
data anslyst.
Please Follow:
Do not add data inspection in the plan, such as "d. oz "print (dF. since this is cheating!
no not use any external information.
encousage yourself print things except final result. More other information lead to be d'atracted.
The code should end-to-end,
Question: (question)
Function
"python [functional plan)
Your entire completion code for function plan executable and correct:
Code:
'python
import pandsa aa pd
Figure 10: Prompt of orchestration coding, This is conducted by SLM Student model.
32

---

To an SOREE scDic ARALELL ond case caioe your catecs mustany cat code tor your core, TH0 peed DACIAAL FAME
araict JENGRINGiA, cigast, Suctiday saates sxincc
begle.
that
steeate: If thep the cortiinaced and cecends incen's conteclitr fecoppss chls your Into natcisile smaline thps
stops stare muttipus cunctoce mustipu closs doe step since Day ai capanist
sntronation sidde cade dais cupplas
internal informctio ond dais complny
1e3Li toey somber shot oelore in decalianes sone stere ursecesaccs and tinee the ixceests seterstandioe of the oseal: logto delece
KunALLAs abapp pepaennte Many cnciesses and Coid appiiacs cuipuss fas the Siten to colecicadd
tkks mush
atiee shaningies ainplait San Sapla, you ton sis calle Mak, siapiity tae fesiine.
eatrrning toe parkuge cietter conctiorally Tale or leto bvem endreseed te eaerlyyn Capue atttert gnttre
cloess actaas clacce oclliaad dook and Zaick 20230ning coreet:
siesss catee
day teppectiio the ontoy "oll heads"" "peire wWrto""
ekouda bo mndetammed you cainel resoscige atubes to paico usker Clllgs eaaps Lse dlas seaall, Nore sidar ixtromatio wald caue skabes
re/etr you
Figure 11: Prompt of plan optimization. This is conducted by LLM Teacher model.
Tou ace date anclyst beadnee, You are ecocating your stodent to penecate pytayes oode to anaver tacolar daca analysta questions.
This suscessful
stody cuse begtet
Data Drecvien st the path (deta_peth) (eleet 20 coMB)2
sysson
(final sechentrated oodel
ecudy! study only
1ed Case Study: [Case Suee]
(csestion)
Table Info: [Dmmacioed Thetal intormation about Tabular Datal dbjaotive
Tlease rote your case stody storld mate your sccdest indecatand. You con't have to incklde code again. Yoo wis2 get 2005 dollars St good
Figure 12: Prompt of case study conversion. This is conducted by LLM Teacher model.
your data atalynia cueations. 20 slicit thair knoxiadge of pandaa you qetteatta atep-by-atep that allow then to fill cods until They accosed.
they
(ca2e
cate endi
isspection code, 'dt.bedd"""
not taes
uxcurnl intornation.
The aised code should be sod-so-end, to you zantot ensourage seudact to peics other things sccept the final rasult, Moze other informatio would causs scodant be distracted.
ginsration.
According to the previoce case atudless yna and reflect how to penerate plara which wake your student Ci31 the cossect oode, dumaerie 5-7 tay points.
Figure 13: Prompt of aggregation prompt of each batch of case studies. This is conducted by LLM Teacher model.
33

---

dsta aralyst tzaine.. You educating acudent data anaiysia cunatiota. To tasz and aileit theis inowledçe of coda, you qenarata atep-by-ssee plase that alloy then. to fill ciegueries uncil they suecedd.
python oods anavar tabulaz
theee atudies
fill
(Jatt layez of case stucinal
o3se end:
Do nos students cad date isapection cost,, auch "4l, "dt.teadd" peiintrt.ineddl iiedd
2o nos guide ties to uae any exteres intornation Lon.
alxed cade abould sid-ta-erd, ao you cannot ancosrage atudent to peint other things excapt the tinsl rasult, endetomend
Polloking case studias, please aurmanice 5-3 bay points about hav to plan and genszata cespect 20d6 to anaeez the tabslsz date anaiyaia conatinta accurabeiyy your diowetay.
to stazt with:
"... You abould
Figure 14: Prompt of summarization prompt of batch of case studies in the last layer. This is conducted by LLM Teacher model.
trainer, teaching student
find
studies:
case becin:
case_arudies)
case ends
Following case studios, ploase aumaariz key 5-7 polnta about hax plan and generate correct python code to the data yssz quest lone
sscceessul plan auquattiona:
Figure 15: Prompt of in-time summarization for meta-instructions. This is conducted by SLM Student model.
You are data enginser. Given the sample data, cenerete pythan code plan to ansvez the question accurately.
Do act add data inspaction in the plan, such ar "df.heac or "printidt. since this is cheating!
not any externel information
encoucage yourself other things excepi finel zesult. information lead distsacted.
Question:
(question)
Thought: to see sarples first zowe:
Code:
hon
import pandes af read eare (data path)) sep=!\ct) print (dr (203
(data. (data_cvcrvics)) cvervion:
thought: should have step-by-step plan for genersting this code first, T xill 611) inco the template in dstails: plan
Steg 1: Stop
Stops
Generate
for questions
let'e think step by step:
"oodee glan Steg
Figure 16: Prompt of generating Chain-OFe Ofain-Of-Thoougtt. This is conducted by SLM Student model.
34

---

the queatioo data uogineer, Siven sasple genecate python
Flease Follow:
Do not cad seta inspaction in the plan, duch as "dthhedd)) or "pxint (at.baadd cince this cheating!
any the pode should be and torwnd, you cennot emoourage yourself to peint otlirr things excepi resalt. Hore other incornation lead to be distracted,
information,
important successtul plan suggestions experts:
"uncossfull plan sungeationa::
[saccessful plan auepeationa]
Question: (question)
Thought: need to see the data sampless in the first 10 rows:
""yython
impert pandes 2s pd
dt pe.zead covl' wwt.c/ccas__paateeeeooooooo "idats_pathi'' seg-"\t") cor. (eedd
Observation:
Thought: Referring [successfol suggestionsl, ahould step-by-step plan generating this code first. will my plan into the terplate detaila: code "odd__ppp plan.
scep 1:
Final Stept
Generate
quastion:
Iet's think stop by step:
code_pian
Step
Figure 17: Prompt of knovledge-driven planning. This conducted by SLM Student model.
enginoox. Civen the sarple dita, gencrata pythor. codo
sacurotely.
Questioo: Iquestionn
Thought: need to see the data parplea in. the fisst 10 zoNG:
Code:
python
pandea
df pd.read esvr' (data [data_pathi] sep-1Net) idn.tdd..boe (b0
Observation:
(data_ [leta_ooeevian)
Thoughts
code this guestion and print the reaslt, will #151 cnde in
texplete:
*1 python
Letts think step by step for the queeztonn: dstep-xiie planst
Code:
* "python
Figure 18: Prompt of code generation given step-wise planning. This is conducted by SLM Student model.
35

---

You are data analyst. Given sasple, you gemerate pandor cade ansver quastion.
Conerace your pandas codo to anavez the questico, and peint the resalt foz your to underscand. Fi11 yeur code in
" "python
(Yeur Codel
Please follow:
not date Inspection in the plan, auch as "dr.head or "print jdf. since this 18 cheating!
not use any asternal information.
The code should end-to-end, you cannat encourage youreel: other the rinal
other infermation would sutdent distracted,
sons exerples:
Examples Star:
[Jov_shot www.ahott_ccpppclll
Examples ENI
Quastion: (question)
Thought:
need
samples it firot
Code:
""pythoo
inport pantasx н* pd de = pd.read o idata_pachh'', sepe"Ser) g ht)
observation:
datta overulee ex)
Thought: cun geoeeaae code to uecaaME this gomesion:
Code:
import pandas
Figure 19: Prompt of few-shot demonstration. This is conducted by SLM Student model.
ARA Case Studl: Average Weight Calculation for Spacific Players
AM Question:
What
Jamarr Sanders and Robert Williams?
Name, Height, Weight (bsi, Position, Hometown, Previous Teamls)
"Sample
Jamanr Sandern: Wright
Robert William:: Weight 210 lbs.
ARa Oejective:
To calculate the everage wright the players lamarr Sanders and Robvrt Willisms from the given datavet.
AM Exploration:
2. "Load Data"": The data loaded from tab separated values (TSV) file.
2. "Fiter Data"": Rows cornesponding the names "ummrr Sanders" and 'Robert Willims' fitered from the dataset.
3. "Calculate Average" The weight of filtered
"Output"" The
fellewing steps,
understand proctical apolication manioulation analysis using Python.
caleudations filtered data, This
Figure 20: Example of case studies for tasks requiring Python. This is conducted by LLM Teacher model.
1. You should ""break down the task into manageable stops"" Each step should build the previsus one, guiding through the process logically.
You should "emphasize data handlnng cleaning" includes handlng missing normallingg and ensuring consistency.
You should "foous filtering extraction"* Guide analysis.
filter extract relevant
spectfic
"perform aggrepation councing" minimam/mm mimmammmmmmmum values derive Insights
appregation
summing, findling
You should "present the result ciearly". Ensure that the final step involves presenting the result in clear and condse monner: This reinforces the importance of commuricating findings effectively."
You should "woid distracsions". " Keep the instructions focused on the end aon--t--nnd process without encouraging unnecessary intermediste outputs or extemal Information. This helps maintain your focus on the task hand."
Figure 21: Example of General Instruction for tasks requiring Python. This is conducted by LLM Teacher model.
36

---

"question":
"which country rank last?"
Understand the problem statement and the data structure.
2. Load the data using appropriate libraries (e.g., pandas).
3. Perform necessary data manipulation and cleaning.
Identify the relevant columns and values for the analysis.
Use appropriate functions methods to filter, sort, and extract
required information.
Output the result in clear and concise manner.
Figure 22: Example of General Instruction for tasks requiring Python. This is conducted by SLM Student model in time.
37


Reviews:
[{"review_id": "P8oyaZDOPB", "reviewer": "Reviewer_xWYB", "summary": "This paper proposes DARGO, a training-free knowledge distillation framework that enhances small language models (SMLs) for data analysis code generation. DARGO is composed of three main components: 1) it constructs solution examples through Model Orchestration Interface (MOI) to orchestrate LLM-SLM collaboration, 2) it filters out correct solutions and condenses them into general reasoning patterns stored in the Memory Database, and 3) it retrieves relevant examples from the Memory Database using RAG for In-Context Learning demonstrations for SMLs. The authors conduct extensive experiments to demonstrate the effectiveness of DARGO.", "questions": "1. The retriever model (i.e., the embedding model) is not perfect. If the retrieved examples are not relevant to the query, could it mislead the SLM and cause a performance drop?\n2. Since the DACG performance of the SML is heavily dependent on the quality and diversity of the Memory Database, does this mean that a large number of examples need to be constructed during the MOI stage? One of the motivations for this paper is to address privacy concerns when using closed-source models. Would requiring large-scale LLM calls during the MOI stage conflict with this motivation?\n3. In the error analysis, 54% of errors are attributed to \"over-reasoning.\" Are there any proposed solutions to this?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The writing is clear, and the motivation is well-defined.\n2. The experiments are thorough, including a wide range of analytical experiments that convincingly prove the effectiveness of DARGO.\n3. The authors have submitted the complete code.\n\nWeaknesses:  \nSee \"Questions\" below.", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: The retriever model (i.e., the embedding model) is not perfect. If the retrieved examples are not relevant to the query, could it mislead the SLM and cause a performance drop?\n\n**A1:** Thank you for this question. We think DARGO is specifically designed to be robust against retrieval noise through several key mechanisms:\n1. DARGO doesn't use raw retrieved cases directly. Instead, it transforms them through **Case Study** translation (Section 2.3), which extracts generalizable reasoning patterns and illustrate how to correctly plan for solving DA tasks rather than specific implementations, making the system inherently robust to individual case quality.\n2. Our meta-instruction (MI) generation process aggregates knowledge across multiple retrieved cases, effectively filtering out noise through consensus-based distillation (Section 2.4).\n3. Our results demonstrate consistent improvements across all datasets and model combinations, indicating that the framework successfully handles retrieval imperfections in practice. Table 10 (Appendix E) presents an ablation focused on embedding and retrieval. Including concrete code examples can bias the SLM (Instruction Type w/ examples), but we mitigate this in DARGO by the distillation of more general and transferrable instructions and case studies.\n4. DARGO introduces novel techniques that mitigate this, but we agree it is not a solved problem. Further promising approaches to reducing such noise would be a RAG implementation by Top K + Top P, which means after retrieval, we should set a value of p as a threshold for filtering those irrelevant.\n\n\n> Q2: Since the DACG performance of the SML is heavily dependent on the quality and diversity of the Memory Database, does this mean that a large number of examples need to be constructed during the MOI stage?\n\n**A2:** Thanks for asking this. We argue that DARGO is a data-efficient knowledge distillation pipeline, without data expansion such techniques as self-evolve, based on the following:\n1. All experiments use modest, fixed datasets (1,000 examples each), demonstrating that DARGO maximizes value from limited data rather than requiring large-scale construction.\n2. The MOI stage refines existing examples and appends SLM-friendly reasoning steps through orchestration and optimization, improving quality rather than increasing quantity.\n3.  As shown in Appendix D.2, DARGO significantly outperforms traditional lora fine-tuning approaches while using the same limited training data, demonstrating its data efficiency. This efficiency is precisely what makes DARGO valuable for DACG domains where high-quality annotated data is inherently scarce. Also, as shown in Table 12, 14, 16, we show that DARGO incurs less costs compared to popular distillation pipelines such as DSPy and ReGAL, while achieving better performance. \n\n> Q3: One of the motivations for this paper is to address privacy concerns when using closed-source models. Would requiring large-scale LLM calls during the MOI stage conflict with this motivation?\n\n**A3:** Thank you for raising this consideration. We believe this design choice strengthens rather than conflicts with our privacy motivation. The key insight is the **separation** between exploration and inference phases. During the exploration phase that includes MOI, memory construction, and case study translation, teacher LLMs work with \"training\" data to build the memory database (similar to how university students attend classes with teachers for learning). These training samples could be acquired from open domains or manually curated in the real-world implementation. In this work, we used small portions of official training split in each benchmark as other previous works did. However, during inference for new queries at test-time, only the SLM operates locally using the pre-constructed memory database, like students taking exams independently using their learned knowledge. This ensures that sensitive user data and queries never interact with external models in testing, maintaining complete privacy during deployment. Once the memory database is constructed, the entire system can operate offline in privacy-sensitive environments without any external dependencies, and this memory database is like a treasure which can be used in multiple downstream tasks. And we prove its generalization in the experiment parts. Further, we also show that DARGO also works with open teacher models (Table 3).\nThanks for asking this.\n\n> Q4: In the error analysis, 54% of errors are attributed to \"over-reasoning.\" Are there any proposed solutions to this?\n\n**A4:** Thank you for this insightful observation and does highlight an opportunity for future work and improvements. In Appendix G, we envision several promising directions to address this. A task complexity router could dynamically determine when reasoning is beneficial versus when direct generation suffices. This router could be implemented as a lightweight classifier or leverage the SLM's own assessment capabilities based on memory database patterns. Additionally, enriching the memory database with diverse examples across different complexity levels may also help. These extensions represent natural evolution paths that could further optimize the reasoning-efficiency trade-off while preserving DARGO's core advantages. We would consider this as a promising direction in the future. Thanks!"}, {"review_id": "SSimPgvDgf", "reviewer": "Reviewer_Nr8J", "summary": "This paper introduces DARGO, a training-free framework for distilling knowledge from Large Language Models (LLMs) to Small Language Models (SLMs) in the context of Data Analysis Code Generation (DACG) tasks. Instead of relying on resource-intensive fine-tuning, DARGO uses In-Context Learning (ICL) to transfer knowledge through a three-phase process: model orchestration, memory collection, and knowledge-guided inference. Experiments on benchmarks such as WIKITQ, TABMWP, and BIRD-SQL show that DARGO significantly improves SLM performance, with an average relative gain of 27.5%. The paper also explores generalization across different model pairings and transfer scenarios, offering a novel and efficient approach to LLM-to-SLM distillation.", "questions": "N/A", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The paper tackles an important and timely problem—efficiently distilling large models into small ones using a training-free approach, which is highly practical.\n2. The framework design is clear and well-structured, with a logical three-phase pipeline.\n3. Empirical results are strong, showing consistent improvements across multiple benchmarks and model architectures.\n\nWeaknesses\n1. The method is engineering-heavy, relying on many handcrafted components, which limits theoretical insight and generalizability.\n2. Comparative experiments are insufficient—there is no evaluation against strong recent code generation models such as Qwen2.5-3B or other modern coder LLMs (at the same setting). Besides, Table 1 only shows the improvement of the proposed method, lacking comparison with other methods that also focus on knowledge distillation.\n3. The presentation quality is poor—several figures are unclear or hard to read, which affects the overall readability of the paper.", "was_revised": true, "final_justification": "The author's rebuttal has addressed my major concerns, so I decided to increase my rating from 2 to 4.", "rebuttal": "> Concern 1: The method is engineering-heavy, relying on many handcrafted components, which limits theoretical insight and generalizability.\n\n**A1:**  Our approach does involve rigorous engineering, however we believe this is essential for real-word challenges in data analysis, and the **application** track under which we have submitted. Our system is intentionally designed, \"with thorough ablation studies validating each module.\" (Reviewer Y9rz). Specifically, we may not agree that this limits the insight and **generalizability** of our work.\n\nTo highlight some key insights from our work:\n\n**MOI, or teacher-student planning**\n\nThrough ablation we show that generated executable plans through iteration between teacher and student is superior to teacher generated plans (Table 5). This is because of a mismatch in granularity between teacher generated plans, and the level of detail required by the student model.  Our work indicates that the student should be involved in the modularization process to correctly align the granularity of the transformation. This is a key insight that we believe is novel and not present in prior work.\n\n**Semantics over syntax**\n\nA further key insight from our works supported via ablation (Table 5) is the role of case studies, or summarizing the demonstrations in a textual and semantic way, rather than being schema, syntax, or example driven. This is especially crucial in the data analysis context where the data is **heterogeneous** and the schema the schema may vary between each task. When compared with other approaches that are syntax based, such as ReGAL (ICML 2024) that generates reusable helper functions, we find that our approach performs better. We believe the concept of case-study summarization is novel in our work, and is rigorously validated across multiple different benchmarks and languages.\n\n**On generalization**\n\nFor generalization, we conduct comprehensive experiments in Section 3.4 demonstrating that our framework successfully transfers across different programming languages (Python, SQL), data modalities (tabular, relational databases), and teacher (GPT-4o, Llama 3.3 70B) /  student models (Llama-3.1-8B, Phi-3-mini, starcoder), as evidenced in our cross-model experiments (Tables 2-3) and both in-domain and out-of-domain evaluations. As part of our response we also show that DARGO generalizes to SLMs with reasoning capabilities. Aside from minor modifications to support different output languages and execution, our system is unchanged across all these configurations, therefore we do not believe that our method inhibits generalizability.\n\nAdditionally, we contend that substantial engineering effort is unavoidable and necessary in the application track. Consider established baselines that have been accepted at top-tier venues: DSPy (ICLR) and ReGAL (ICML), both of which inherently contain handcrafted components. Compared to these methods, our approach actually requires **less** engineering overhead while being more **cost-effective**, as demonstrated by our comprehensive cost analysis across all procedures in Appendix I, where DARGO proves to be the most resource-efficient solution.\n\n> Concern 2: Comparative experiments are insufficient—there is no evaluation against strong recent code generation models such as Qwen2.5-3B or other modern coder LLMs (at the same setting).\n\n**A2:** We appreciate the suggestion to include more specific models, including code generating and reasoning models. The nature of the field is such that new models will always be released post-submission. Our primary contribution is demonstrating effective training-free knowledge distillation, as acknowledged by Reviewer xWYB, rather than showcasing specific model performance. \n\nWe show consistent improvements across diverse architectures that were current at the time of submission, validating our approach's robustness. Concretely, for teacher models, we conduct the same auto pipeline for GPT-4o and Llama 3.3 70B; for student models, we test Llama 3 8B, phi-3-mini, starcoder 7B. Additionally, we provide extensive ablation studies (Section 3.5) and out-of-distribution evaluations (Section 3.4) that demonstrate the method's effectiveness beyond simple performance comparisons. \n\nHowever, we appreciate your suggestions for experiments for very recent SLMs, and acknowledge that reasoning models are interesting to study. Here we evaluate DarGO on Qwen 2.5 Coder in 1.5B and 7B, and a very recent reasoning model Qwen-3-8B with reasoning enabled. Results:\n\n| Model (Qwen2.5-Coder-1.5B) | BIRD-SQL | WIKITQ | TabMWP | CRTQA | QrData | INFI-AGENT |\n|---------------------------|----------|--------|--------|-------|--------|------------|\n| Baseline                 | 22.4     | 28.5   | 35.8   | 25.27 | 43.29  | 42.8       |\n| + DarGO                     | 30.2     | 35.2   | 48.5   | 42.86 | 48.61  | 46.3       |\n\n| Model (Qwen2.5-Coder-7B) | BIRD-SQL | WIKITQ | TabMWP | CRTQA | QrData | INFI-AGENT |\n|--------------------------|----------|--------|--------|-------|--------|------------|\n| Baseline               | 37.2     | 36.8   | 41.2   | 34.75 | 48.86  | 51.36      |\n| + DarGO                    | 43.4     | 44.1   | 52.8   | 43.82 | 50.13  | 54.47      |\n\n| Model (Qwen3-8B) | Mini-Dev | WIKITQ | TabMWP | CRTQA | QrData | INFI-AGENT |\n|------------------|----------|--------|--------|-------|--------|------------|\n|Baseline        | 41       | 37.5   | 45.1   | 40.52 | 44.05  | 47.13      |\n| +DarGO            | 47.2     | 42.8   | 48.8   | 46.7  | 48.86  | 53.7       |\n\nThe consistent performance improvements across these recent models demonstrate that DARGO generalizes effectively, including specialized code generation models and reasoning-enhanced variants, further validating the statement of the paper.\n\n> Concern 3: Besides, Table 1 only shows the improvement of the proposed method, lacking comparison with other methods that also focus on knowledge distillation.\n\n**A3:** We do think there are some misunderstandings for this concern. **In the 4th and 5th line of SLM performance in Table 1**, we compare our performance with one popular distillation pipeline **DSPy** (ICLR 2024) for general NLP tasks, a function-based distillation pipeline for code generation **ReGAL** (ICML 2024).\n\nSection 3.3 (2) gives a comparison with these approaches and we further present a cost analysis in Appendix I. Our results show that DARGO is more effective and cost efficient. A key insight from our analysis is that DARGO is more robust due to its distillation approach. DSPy is few-shot based that can overfit or be effected by noisy data. ReGAL generates reusable python helper functions, however these are tightly coupled to specific domains and schema (e.g. `filter_senator_by_year`). Our contribution of case-study summarization and instruction based distillation mitigates these issues.\n\n> Concern 4: The presentation quality is poor—several figures are unclear or hard to read, which affects the overall readability of the paper.\n\n**A4:** Thanks for raising this. We appreciate that some figures could benefit from enhancement and clearer graphics, and will address this in revision."}, {"review_id": "ReP6Nons8n", "reviewer": "Reviewer_nVTU", "summary": "The paper introduces a training free knowledge distillation approach to make SLMs better at data analysis code generation tasks. They achieve this through the curation of synthetically generated functional plans from a teacher model that are adapted to the student. These plans are used as context during inference with the help of RAG. It evaluates this approach on multiple publicly available datasets and models.", "questions": "- In the results shown in Table 9, is the LoRA model fine-tuned on the original data, or on the constructed database that is used during RAG?\n- The result tables in the main paper are missing standard errors. Is this data included elsewhere?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The approach to use the synthetically generated data in-context without fine-tuning is quite useful and avoids expensive training runs.\n- MOI (Section 2.2) and the ablation experiment from Table 5 show the effectiveness of SLM participation during the planning phase.\n- The paper also shows the generalization of the generated knowledge by using it with other student models (Section 3.4).\n\nWeaknesses:\n- The experiments mainly focus on non-reasoning SLMs like Phi-3, but omit recent reasoning specialized SLMs. Gain in results for such models might differ, since they are already trained for task decomposition in general purpose scenarios.\n- Since the paper focuses on a training-free approach, it would be useful to see a proper comparison between fine-tuning and RAG, by using the same constructed memory database for fine-tuning. \n- Evaluation scope is limited to data analysis tasks. This approach is general enough that it could work with a broader set of tasks.\n- The discussion on the broader impact and future work would be more useful if moved to the main paper.", "was_revised": true, "final_justification": "Author rebuttal addressed most of my concerns. The overall pipeline shows benefits with ablations to justify each component. My criticisms for this paper:\n1. Evaluating on more varied domains will strengthen the paper if successful. Nothing in the pipeline is very specific to DACG.\n2. As mentioned by another review, I agree that the method is a bit engineering heavy.", "rebuttal": "> Concern 1: The experiments mainly focus on non-reasoning SLMs like Phi-3, but omit recent reasoning specialized SLMs. Gain in results for such models might differ, since they are already trained for task decomposition in general purpose scenarios.\n\n**A1:** Thanks for raising this. The release of many of these models, such as Qwen-3 with reasoning capability, coincided with submission of this work. (Qwen-3 released 14th May, and Phi-4-Reasoning appeared on 30th April). We do agree that reasoning SLMs may exhibit different interactions and are important to consider. As a result, we conduct additional experiments of all settings for qwen-3-8B with reasoning enabled as:\n\n| Model (Qwen3-8B) | BIRD-SQL | WIKITQ | TabMWP | CRTQA | QrData | INFI-AGENT |\n|------------------|----------|--------|--------|-------|--------|------------|\n| Zero-shot        | 41       | 37.5   | 45.1   | 40.52 | 44.05  | 47.13      |\n| + DarGO           | 47.2     | 42.8   | 48.8   | 46.7  | 48.86  | 53.7       |\n\nAs we can observe, our method can be successfully applied to recent reasoning based SLMs with no additional modification of the pipeline. The performance of Qwen3-8B with reasoning enabled powered by DarGO is the best on BIRD-SQL (Table 1) with 47.2%. We thank for your suggestions and curiosity, we will add them into revision if possible.\n\n> Concern 2: In the results shown in Table 9, is the LoRA model fine-tuned on the original data, or on the constructed database that is used during RAG? Since the paper focuses on a training-free approach, it would be useful to see a proper comparison between fine-tuning and RAG, by using the same constructed memory database for fine-tuning.\n\n**A2:** Thanks for asking this. We already have comparison with RAG based approach in which dynamic few shot in Table 1 was implemented by retrieving relevant examples from our output memory database as few shot examples for augmentation of generation. The FT results in Table 9 are based on same initial training samples, not the samples from the memory database since we considered **auto-generated memory database** as our contribution. However, we agree that SFT on the memory database would strengthen our work. We present experiment and list result here:\n| Model | SIMPLE | MEDIUM | CHALLENGING | OVERALL |\n|-------|--------|--------|-------------|---------|\n| | ***Zero-Shot End-to-End Code Gen.*** | | | |\n| Original Checkpoint | 38.51 | 21.20 | 11.76 | 24.40 |\n| LoRA Fine-Tuned | 39.86 | 19.20 | 10.78 | 23.60 |\n| SFT MEM | 40.12 | 22.80 | 12.45 | 27.20 |\n| | ***DARGO Knowledge Distillation*** | | | |\n| Meta Instruction | 51.35 | 30.40 | 16.67 | 33.80 |\n\nThese results demonstrate that our memory database provides substantial benefits even for traditional fine-tuning approaches. We attribute this improvement to two key factors:\n  1. our Model Orchestration Interface (MOI) effectively filters noisy or low-quality data, including ground truth errors that may mislead training\n  2. the orchestrated codes in our memory database contain structured comments and planning information that enhance reasoning capabilities, contrasting with original declarative ground truth codes that may impose additional inferential burden on SLMs during learning.\n\nAs we illustrate in D.2, we hypothesize that fine-tuning high quality data may exceed ICL when the samples are large enough.\n\nNotably, while SFT on our memory database shows improvement over baseline fine-tuning, our training-free knowledge distillation approach still achieves superior performance while avoiding the computational overhead of parameter updates.\n\n> Concern 3: Evaluation scope is limited to data analysis tasks. This approach is general enough that it could work with a broader set of tasks.\n\n**A3:** Thanks for your suggestion and acknowledgement of the generality of this approach. We agree that it can be applied to a broader set of tasks, however we focus on data analysis code generation (DACG) due to its significance and challenge. By focusing on DACG we can provide more in-depth analysis and comparison with other approaches. Specifically:\n\n1) DACG represents a complex and representative domain that requires both code generation capability and data understanding capability, making it a rich testbed for knowledge distillation beyond syntax. Success in DACG demonstrates the ability of our work to handle tasks requiring multi-step reasoning and domain-specific expertise.\n\n2) DACG has a significant real-world impact and ubiquity. Data analysis and code generation are crucial needs across industries and research. Improving DACG directly benefits data scientists, researchers, and analysts in their daily work especially since the data covers a wide range of domains such as analysis in biomedical science, sports, finance, education.\n\n3) DACG introduces unique challenges are not as prevalent in other code generation domains. For example, a key challenge that we found and addressed was handling heterogeneous data contexts and schema during distillation. If the distillation was too specific to the schema, it would not generalize, confusing the SLM during inference. Existing work that distils helper functions (e.g. ReGAL) created functions that were coupled too specific to domains and thus not generalizable. By focusing on DACG we are able to explore the challenges in depth.\n\n> Concern 4: The discussion on the broader impact and future work would be more useful if moved to the main paper.\n\n**A4:** Thanks for your suggestion, we will do this in revision.\n\n> Concern 5: The result tables in the main paper are missing standard errors. Is this data included elsewhere?\n\n**A5:** Thanks for asking this. For each result, we conduct experiment 5 times and report averaged results for neat presentation in tables as we mentioned in Checklist Section 7 (L. 730-732) with temperature as 0.0 and set top_p as 1 so the performance is stable. But we also summarize this statistics for methods across datasets here:\n\n| Dataset | Baseline Methods | DARGO Methods |\n|---------|------------------|---------------|\n| **BIRD-SQL** | ±0.3 | ±0.1 |\n| **WikiTQ** | ±1.5 | ±1.1 |\n| **TabMWP** | ±1.3 | ±0.9 |\n| **CRTQA** | ±1.2 | ±0.8 |\n| **QrData** | ±0.1 | ±0.2 |\n| **INFI-AGENT** | ±0.8 | ±0.3 |"}, {"review_id": "Hf8pWSRhVu", "reviewer": "Reviewer_Y9rz", "summary": "This work asks: Can we transfer an LLM’s reasoning ability to a smaller, locally deployable student language model (SLM) without any parameter updates? To address this, the authors propose a non-parametric knowledge distillation framework that requires no fine-tuning, delivering LLM-level DACG capabilities on compact SLMs.\nIt introduces DARGO, a three-stage pipeline including abstraction lifting, orchestration coding, and plan optimization.\nDARGO is evaluated on three challenging benchmarks, WIKITQ, TABMWP, and BIRD-SQL.", "questions": "1. How do you define small language model and large language model? How to distinguish them?\n2. Which BIRD setting are you using? Is `Oracle Knowledge` provided to the model? This is a huge difference.", "limitations": "1.  Repeated teacher calls is still not cheap. These multi-round executions may be impractical in low-budget or real-time settings.\n2.  Large memory banks could slow down the retrieval part.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. Proposed a non-parametric, in-context distillation from LLM to SLM, obviating costly fine-tuning.\n2. Covers planning (MOI), bank construction, and inference (RAG-MI), with thorough ablation studies validating each module.\n3. Large improvements on multiple benchmarks using small models.\n\nWeaknesses\n1. The MOI phase demands multiple teacher-model calls and code executions, which may be expensive.\n2. As task volume grows, retrieval latency and storage demands could become bottlenecks.\n3. Focuses exclusively on Python/SQL for tabular data tasks, while other languages or code-generation scenarios remain unexplored.\n4. Relying on closed-source LLMs as teachers still remain privacy and access concerns.", "was_revised": true, "final_justification": "I will keep my original score. The authors’ initial response was somewhat evasive, offering common-sense statements rather than clarifying their claims in the paper. Moreover, the overall idea of this work is not novel, and the use of an oracle setting on BIRD in the experiments makes it mediocre rather than demonstrating strong reasoning and generalization capabilities.", "rebuttal": "> Concern 1: The MOI phase demands multiple teacher-model calls and code executions, which may be expensive.\n\n**A1:** Thanks for this concern. We think our method provides a cost-effective knowledge distillation pipeline. We computed a detailed cost analysis in Appendix I with Table 12, Table 19, and Table 21. We show that our method is cheaper compared to popular knowledge distillation approaches DSPy (for general NLP tasks) and ReGAL (specifically for code tasks) whilst providing better performance.\n\n> Concern 2: As task volume grows, retrieval latency and storage demands could become bottlenecks.\n\n**A2:** Thank you for raising this important concern. We believe there may be some misunderstanding about our workflow, so please allow us to clarify the operational phases and scalability characteristics of DARGO:\n\n**On retrieval latency:**\nOur framework operates in two distinct phases: **exploration** and **inference**. During exploration, we construct a memory database by storing successful cases generated through LLM-SLM orchestration on the training set. Crucially, once exploration is complete, this database becomes **frozen** and remains static throughout all following inference operations for unseen tasks in testing. During inference, each unseen testing tasks follows an identical workflow: retrieve exactly 3 case studies from the fixed database → synthesize meta-instructions → generate code. This design ensures that inference latency is **orthogonal** to task volume growth because the database size, retrieval count, and processing workflow remain constant regardless of the number of evaluation testing examples processed. It will be the case that retrieval latency increases as the number of cases in the database increases as there are more entries to consider, but this is true of any RAG-based system that employs a vector database. We do not expect this to be a bottleneck due to its efficient indexing, especially when compared to SLM inference latency of any system.\n\n**On storage demands:**\n1) As discussed in Section 1, high-quality training data for DACG is inherently scarce. Even in our largest experimental setting (1,000 training samples), our exploration phase retains only successful cases, resulting in compact memory databases. Despite this low-resource constraint, our experiments demonstrate encouraging performance improvements.\n2) For adopting our method with larger datasets, scalability can be easily managed by implementing a maximum memory database size limit, ensuring consistent retrieval latency regardless of training set scale.\n\nNotably, we maintain strict experimental integrity by **NEVER** incorporating test cases into the memory database during inference, preventing any unfair advantage from ground truth leakage while ensuring fair comparative evaluation. Therefore, regardless of the volume of inference tasks processed, latency and storage concerns remain constant and do not pose scalability bottlenecks.\n\n> Concern 3: Focuses exclusively on Python/SQL for tabular data tasks, while other languages or code-generation scenarios remain unexplored.\n\n**A3:** Thanks for your suggestion. We provide our response here:\n\n1) DACG represents a complex and representative domain that requires both code generation capability and data understanding capability, making it a rich testbed for knowledge distillation beyond syntax. Success in DACG demonstrates the ability of our work to handle tasks requiring multi-step reasoning and domain-specific expertise.\n\n2) DACG has a significant real-world impact and ubiquity. Data analysis and code generation are crucial needs across industries and research. Improving DACG directly benefits data scientists, researchers, and analysts in their daily work especially since the data covers a wide range of domains such as analysis in biomedical science, sports, finance, education.\n\n3) DACG introduces unique challenges are not as prevalent in other code generation domains. For example, a key challenge that we found and addressed was handling heterogeneous data contexts and schema during distillation. If the distillation was too specific to the schema, it would not generalize, confusing the SLM during inference. Existing work that distils helper functions (e.g. ReGAL) created functions that were coupled too specific to domains and thus not generalizable. By focusing on DACG we are able to explore the challenges in depth.\n\nFurther, we summarized from open survey to prove research about Python and SQL can cover most usage in the real world:\n1. Stack Overflow's 2024 Developer Survey (65,000+ respondents) shows that 73% of data professionals use Python regularly and 60% regularly use SQL.\n2. Industry reports indicate that over 85% of data job postings mention SQL as a must-have skill 70% of data roles demand SQL knowledge according to DataCamp surveys.\n\n> Concern 4: Relying on closed-source LLMs as teachers still remain privacy and access concerns.\n\n**A4:** Thanks for asking. However, we think there are some misunderstanding. We believe this design choice actually strengthens rather than conflicts with our privacy motivation.\n\n1)  The key insight is the separation between exploration and inference phases. During the exploration including MOI stage, memory construction, and case study translation, teacher LLMs work with \"training\" data to build the memory database (similar to how university students attend classes with teachers for learning). However, during inference for new queries in testing, only the SLM operates locally using the pre-constructed memory database, like students taking exams independently using their learned knowledge. This ensures that sensitive user data and queries **NEVER** interact with external models in testing, maintaining complete privacy during deployment. Once the memory database is constructed, the entire system can operate offline in privacy-sensitive environments without any external dependencies.\n\n2) Additionally, we show the performance of DarGO when the teacher LLM is replaced with an open-weight and popular model **Llama 3.3 70B**. Section 3.4 (paragraph) with the evidence in Table. 3, highlighting that DarGO remains effective with purely open models. This supports our claim that we are proposing an effective knowledge distillation workflow, instead of relying on specific proprietary models on both teacher and student ends.\n\n> Question 1: How do you define small language model and large language model? How to distinguish them?\n\n**A5:** Thank you for this important clarification question.\nHere are our operational definition of Small Language Models (SLMs):\nWe define SLMs based on two key criteria mentioned in Lines 207-210:\n\n1) Computational constraint: Models with ≤15B parameters that can be efficiently deployed on a single high-end GPU (A100/A800). This threshold reflects practical deployment considerations for resource-constrained environments.\n2) Emergent capability (ICL) threshold: Models must demonstrate in-context learning (ICL) capabilities, including instruction following and few-shot adaptation without requiring supervised fine-tuning for new tasks. In such context, models like T5 or BERT, despite having fewer parameters, lack the instruction-following and ICL capabilities we associate with modern language models, and thus fall outside our SLM definition. \n\n> Question 2: Which BIRD setting are you using? Is Oracle Knowledge provided to the model? This is a huge difference.\n\n**A2:** Thanks for asking this. Yes we use Oracle Knowledge by appending it to question as part of user task query which is a common or default usage for this dataset. Observing BIRD-SQL leaderboard, almost all works include Oracle Knowledge into input. But we will add this detail in revision if we get a chance. Thanks for highlighting this and improving the rigor of the paper. Thank you!"}]

Author rebuttal:

Meta-review: ### **The Key contribution of the paper:**
- **A Novel Framework for Knowledge Distillation**: The paper introduces DARGO (Distillation and Adaptive Reasoning-Guided Orchestration), a novel framework designed to transfer knowledge from a powerful Large Language Model (LLM) "teacher" to a more efficient Small Language Model (SLM) "student" without the need for intensive fine-tuning. The framework is particularly useful for Data Analysis Code Generation (DACG) tasks, where privacy is a concern.

- **Effective Use of In-Context Learning (ICL)**: It proposes a non-parametric knowledge distillation method that uses In-Context Learning (ICL), which is a training-free approach, to adapt SLMs for complex tasks. This approach is a significant alternative to traditional, resource-intensive fine-tuning, which often faces challenges due to limited high-quality training data and poor generalization across different programming languages or updates.

### **Summarization of Reviewer Comments**
The reviews for Submission 21305 are largely positive, with all three reviewers recommending acceptance. Reviewer xWYB initially provided a high score, while Reviewers Nr8J and nVTU were initially more reserved, citing concerns about experimental scope, engineering complexity, and presentation. The authors' detailed and data-driven rebuttal successfully addressed these weaknesses by providing clarifications, additional experimental results on recent models, and robust defenses of their design choices. The new data, which showed DARGO's consistent improvements on Qwen and other models, was particularly effective in swaying the reviewers. The overall consensus is that the paper presents a technically solid, valuable, and practical framework with strong empirical evidence, and the authors effectively handled the criticisms raised during the review period.

**For the response to Reviewer Y9rz**:
The authors provided a clear rebuttal to the reviewer's key concerns:
- **Cost**: The authors demonstrated that their DARGO framework is more cost-effective and delivers better performance compared to other distillation methods like DSPy and ReGAL.

- **Speed and Storage**: The reviewer raised concerns about potential bottlenecks from increased task volume. The authors clarified that the system's database is fixed during inference, ensuring that the per-query latency remains constant regardless of the total number of test queries. Although the reviewer dismissed this as "common sense," the authors' explanation confirms the system's speed advantage in its design.

- **Scope**: To justify their focus on Python/SQL, the authors highlighted the complexity and real-world significance of Data Analysis Code Generation (DACG), citing developer survey data to support the relevance and impact of their work.

In summary, the authors’ rebuttal effectively addresses the reviewer's technical criticisms by providing concrete evidence of their system's efficiency and a strong rationale for their research focus. While the reviewer remained unconvinced, the authors' explanations validate the paper's technical merits.
Decision: Accept (poster)