Sky-06/qwen2.5-coder-1.5b-agentic-sft

Qwen2.5-Coder-1.5B — agentic + code SFT

A 1.5B model fine-tuned for code generation with tool-calling / agentic behavior. Base model: Qwen/Qwen2.5-Coder-1.5B-Instruct (Apache-2.0). Trained with TRL SFTTrainer in one epoch over a curated HuggingFaceTB/smoltalk mix, following the SmolLM2 small-model recipe (arXiv:2502.02737).

Evaluation (greedy, harness sanity-verified 164/164 on canonical solutions)

Model	HumanEval pass@1	Tool-call format	Valid tool name	Valid arg types
this model	62.2% (102/164)	96%	96%	96%
base Qwen2.5-Coder-1.5B-Instruct	65.2% (107/164)	0%	—	—

Tool-call metrics are on 300 held-out rows of smoltalk apigen-80k test split (never trained on): does the reply contain a parseable `` block with tools named in the system prompt and dict-typed arguments.

Interpretation: SFT traded ~3 points of raw code ability for tool-calling compliance the base model entirely lacks.

Training data (102,260 rows, messages format)

Source	Rows	Teaches
smoltalk apigen-80k	25,000	tool-calling traces: <tools> schemas in system prompt, `` responses, multi-turn
smoltalk self-oss-instruct	30,000	Python code instruction-following with test assertions
smoltalk smol-magpie-ultra	25,000	general multi-turn chat quality
smoltalk metamathqa-50k	10,000	math reasoning
smoltalk smol-summarize / smol-rewrite / everyday-conversations	12,260	instruction-following breadth

Training details

• Method: full fine-tune, SFT with packing (bfd, 1024 tokens/block, flash-attn2 varlen via prebuilt kernels-community/flash-attn2), Liger fused CE, loss over full sequence (SmolLM2-style)
• Hyperparameters: lr 2e-5 cosine (warmup 48 steps), global batch 64 (8 × 8 accum), bf16, AdamW fused, 1 epoch = 852 steps / 55.2M tokens
• Hardware: 1× A100-80GB (HF Jobs), 62.5 min at 4.25 s/step
• Stack: TRL 1.15.0 · Transformers 5.19.0 · PyTorch 2.14.1

Usage

Tool-calling format (as in apigen-80k): pass tool schemas in the system prompt inside <tools>...</tools>; the model answers with `` blocks, matching the Qwen chat template shipped in this repo.

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Sky-06/qwen2.5-coder-1.5b-agentic-sft", device_map="auto")
tok = AutoTokenizer.from_pretrained("Sky-06/qwen2.5-coder-1.5b-agentic-sft")
messages = [{"role": "user", "content": "Write a Python function that merges two sorted lists."}]
print(tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False))

Limitations

• 1.5B model: limited multi-step agentic reliability; long tool chains will often derail
• Trained on synthetic teacher data (Llama-3.1-70B / self-instruct); inherits its biases
• Tool traces use a single canonical call format; other formats (Hermes, ToolACE) were not mixed
• HumanEval measured greedy pass@1 with a custom exec harness (sanity-checked on canonical solutions), not the official evaluation suite

Citations

SmolTalk / SmolLM2 recipe:

@article{allal2025smollm2,
  title   = {{SmolLM2}: When Smol Goes Big -- Data-Centric Training of a Small Language Model},
  author  = {Allal, Loubna Ben and Lozhkov, Anton and Bakouch, Elie and others},
  journal = {arXiv preprint arXiv:2502.02737},
  year    = {2025}
}

Qwen2.5-Coder base:

@article{hui2024qwen2,
  title   = {{Qwen2.5-Coder} Technical Report},
  author  = {Hui, Jian and Yang, Kaichun and others},
  journal = {arXiv preprint arXiv:2409.12186},
  year    = {2024}
}

TRL:

@software{vonwerra2020trl,
  title   = {{TRL: Transformers Reinforcement Learning}},
  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and others},
  license = {Apache-2.0},
  url     = {https://github.com/huggingface/trl},
  year    = {2020}
}