Title: Unlocking SLM Potential for Data Analysis Code Generation via Non-Parametric Knowledge Distillation
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

Knowledge Distillation

Data Code Generation
Abstract: Knowledge distillation from Large Language Models (LLMs) to locally hosted Small Language Models (SLMs) provides advantages for Data Analysis Code Generation (DACG) such as privacy protection. However, achieving effective distillation without resource-intensive training is challenging. This paper investigates whether LLMs can distill knowledge to SLMs through In-Context Learning (ICL), a training-free method for rapid task adaptation. We present the DarGO: Distillation and Adaptive Reasoning-Guided Orchestration framework, which facilitates automatic knowledge distillation from LLMs to SLMs. DarGO consists of three phases: exploration through an Model Orchestration Interface (MOI), Memory Collection of successful trajectories, and Knoweldge-driven Inference. We evaluate DarGO on three challenging DACG benchmarks (WikiTQ, TabMWP, and Bird-SQL), each with in-domain training sets that enable detailed analysis of knowledge distillation effectiveness. DarGO demonstrates a substantial relative performance improvement of 27.5\% on average for the student SLMs. To further observe generalization capabilities, we evaluate the \method across different teacher-student model combinations, knowledge transfer scenarios, and unified memory approaches for more advanced, test-only data analysis tasks. Our findings contribute a novel perspective on distillation methods that enhance high performance for SLMs while avoiding intensive fine-tuning.


Peer reviews:
comments: SUMMARY:
This work asks: Can we transfer an LLM’s reasoning ability to a smaller, locally deployable student language model (SLM) without any parameter updates? To address this, the authors propose a non-parametric knowledge distillation framework that requires no fine-tuning, delivering LLM-level DACG capabilities on compact SLMs.
It introduces DARGO, a three-stage pipeline including abstraction lifting, orchestration coding, and plan optimization.
DARGO is evaluated on three challenging benchmarks, WIKITQ, TABMWP, and BIRD-SQL.

STRENGTHS AND WEAKNESSES:
Strengths
1. Proposed a non-parametric, in-context distillation from LLM to SLM, obviating costly fine-tuning.
2. Covers planning (MOI), bank construction, and inference (RAG-MI), with thorough ablation studies validating each module.
3. Large improvements on multiple benchmarks using small models.

Weaknesses
1. The MOI phase demands multiple teacher-model calls and code executions, which may be expensive.
2. As task volume grows, retrieval latency and storage demands could become bottlenecks.
3. Focuses exclusively on Python/SQL for tabular data tasks, while other languages or code-generation scenarios remain unexplored.
4. Relying on closed-source LLMs as teachers still remain privacy and access concerns.

QUESTIONS:
1. How do you define small language model and large language model? How to distinguish them?
2. Which BIRD setting are you using? Is `Oracle Knowledge` provided to the model? This is a huge difference.

LIMITATIONS:
1.  Repeated teacher calls is still not cheap. These multi-round executions may be impractical in low-budget or real-time settings.
2.  Large memory banks could slow down the retrieval part.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a training free knowledge distillation approach to make SLMs better at data analysis code generation tasks. They achieve this through the curation of synthetically generated functional plans from a teacher model that are adapted to the student. These plans are used as context during inference with the help of RAG. It evaluates this approach on multiple publicly available datasets and models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The approach to use the synthetically generated data in-context without fine-tuning is quite useful and avoids expensive training runs.\n- MOI (Section 2.2) and the ablation experiment from Table 5 show the effectiveness of SLM participation during the planning phase.\n- The paper also shows the generalization of the generated knowledge by using it with other student models (Section 3.4).\n\nWeaknesses:\n- The experiments mainly focus on non-reasoning SLMs like Phi-3, but omit recent reasoning specialized SLMs. Gain in results for such models might differ, since they are already trained for task decomposition in general purpose scenarios.\n- Since the paper focuses on a training-free approach, it would be useful to see a proper comparison between fine-tuning and RAG, by using the same constructed memory database for fine-tuning. \n- Evaluation scope is limited to data analysis tasks. This approach is general enough that it could work with a broader set of tasks.\n- The discussion on the broader impact and future work would be more useful if moved to the main paper.\n\nQUESTIONS:\n- In the results shown in Table 9, is the LoRA model fine-tuned on the original data, or on the constructed database that is used during RAG?\n- The result tables in the main paper are missing standard errors. Is this data included elsewhere?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DARGO, a training-free framework for distilling knowledge from Large Language Models (LLMs) to Small Language Models (SLMs) in the context of Data Analysis Code Generation (DACG) tasks. Instead of relying on resource-intensive fine-tuning, DARGO uses In-Context Learning (ICL) to transfer knowledge through a three-phase process: model orchestration, memory collection, and knowledge-guided inference. Experiments on benchmarks such as WIKITQ, TABMWP, and BIRD-SQL show that DARGO significantly improves SLM performance, with an average relative gain of 27.5%. The paper also explores generalization across different model pairings and transfer scenarios, offering a novel and efficient approach to LLM-to-SLM distillation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper tackles an important and timely problem—efficiently distilling large models into small ones using a training-free approach, which is highly practical.\n2. The framework design is clear and well-structured, with a logical three-phase pipeline.\n3. Empirical results are strong, showing consistent improvements across multiple benchmarks and model architectures.\n\nWeaknesses\n1. The method is engineering-heavy, relying on many handcrafted components, which limits theoretical insight and generalizability.\n2. Comparative experiments are insufficient—there is no evaluation against strong recent code generation models such as Qwen2.5-3B or other modern coder LLMs (at the same setting). Besides, Table 1 only shows the improvement of the proposed method, lacking comparison with other methods that also focus on knowledge distillation.\n3. The presentation quality is poor—several figures are unclear or hard to read, which affects the overall readability of the paper.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper proposes DARGO, a training-free knowledge distillation framework that enhances small language models (SMLs) for data analysis code generation. DARGO is composed of three main components: 1) it constructs solution examples through Model Orchestration Interface (MOI) to orchestrate LLM-SLM collaboration, 2) it filters out correct solutions and condenses them into general reasoning patterns stored in the Memory Database, and 3) it retrieves relevant examples from the Memory Database using RAG for In-Context Learning demonstrations for SMLs. The authors conduct extensive experiments to demonstrate the effectiveness of DARGO.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The writing is clear, and the motivation is well-defined.
2. The experiments are thorough, including a wide range of analytical experiments that convincingly prove the effectiveness of DARGO.
3. The authors have submitted the complete code.

Weaknesses:  
See "Questions" below.

QUESTIONS:
1. The retriever model (i.e., the embedding model) is not perfect. If the retrieved examples are not relevant to the query, could it mislead the SLM and cause a performance drop?
2. Since the DACG performance of the SML is heavily dependent on the quality and diversity of the Memory Database, does this mean that a large number of examples need to be constructed during the MOI stage? One of the motivations for this paper is to address privacy concerns when using closed-source models. Would requiring large-scale LLM calls during the MOI stage conflict with this motivation?
3. In the error analysis, 54% of errors are attributed to "over-reasoning." Are there any proposed solutions to this?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03