OpenGrad — Qwen3.5-2B, M0 SFT on canonical corpus v2

Full-parameter supervised fine-tuning of Qwen/Qwen3.5-2B for tool-calling decision boundaries, produced by OpenGrad.

These are research checkpoints, not production models. They are published because the experiment behind them is unusual: the same training run was performed twice on two corpora that differ in exactly one respect, and the two results disagree completely. That contrast is the artefact of interest.

Summary

The experiment tests whether a tool-calling collapse observed during fine-tuning is caused by the training procedure or by the training corpus. The same trainer, hyperparameters, base checkpoint, renderer, and evaluation were applied to two corpora:

corpus	trainable records	records whose target contains a tool call
v1 (published)	55,719 of 213,951	9 (0.0162%)
v2 (this model)	101,785 of 103,036	48,723 (47.9%)

On v1 the models learn to stop calling tools entirely. On v2 they learn the decision boundary. The corpus composition, not the training procedure, was the cause.

Checkpoints

checkpoint	call_f1	precision	recall	over-call	macro†
`checkpoint-600`	0.5672	0.7450	0.4579	0.0862	0.6242
`checkpoint-1200`	0.5995	0.7373	0.5093	0.0989	—
`checkpoint-1800`	0.5247	0.7891	0.3942	0.0577	—
`checkpoint-2400`	see below
B0 baseline	0.6191	0.4542	0.9722	0.6425	0.3621

† macro = mean per-class recall over CALL / CLARIFY / UNSUPPORTED.

Measured on a frozen 3,650-example behavioural held-out set with vLLM 0.29.0 at temperature 0, using the same renderer, parser, and generation settings that produced the baseline.

How to read this

call_f1 alone is misleading here, and that is part of the finding.

The baseline B0 scores 0.6191 on call_f1 by calling a tool on 64.3% of examples whose correct answer is not a call. It recalls 97.2% of gold CALLs and 1.3% of gold UNSUPPORTEDs — it has not learned a decision boundary, it has learned to always call. On a balanced three-class view B0 is the worst model in this table (macro 0.3621) and every checkpoint here is better.

So:

checkpoint-600 is the most balanced model measured. It is better than B0 on call_precision (0.745 vs 0.454), better on over-calling (0.086 vs 0.643), better on macro recall (0.624 vs 0.362), and worse on call_f1 (0.567 vs 0.619) and recall (0.458 vs 0.972).
None of these is a finished tool-calling model. A macro of 0.62 with 46% call recall is a model that has begun to learn a boundary, not one that has mastered it.
checkpoint-1200 has the highest call_f1; checkpoint-1800 the highest precision and lowest over-calling; checkpoint-2400 the lowest training loss. Choosing among them on this metric is selecting on the evaluation set, so treat the ranking as diagnostic rather than as a result.

Training


Base model	`Qwen/Qwen3.5-2B` @ `15852e8c16360a2fea060d615a32b45270f8a8fc`
Tokenizer	same revision
Method	full-parameter SFT (100% of 1,881,825,088 parameters)
Precision	bfloat16, gradient checkpointing
Optimizer	AdamW, cosine schedule, 120-step warmup, grad-clip 1.0
Steps	2,400
Effective batch	16 (8 x 2 accumulation), bounded at 4,096 tokens per micro-batch
LR	1e-5
Sequence window	2,048 tokens
Seen	27,672 examples / 8,011,435 supervised tokens
Loss	1.5332 → 0.5582 (mean 0.4618)
Hardware	1x A100-SXM4-80GB, 42 min, 28.1 GiB peak

Loss is computed only on assistant-turn tokens. The pinned Qwen chat template is position-dependent — it emits a <think> block only for an assistant turn after the last real user query — so target spans are located by character offset and mapped back through the tokenizer's offset_mapping, with each span self-checked against the rendered text. See docs/SFT_TRAINING.md in the repository.

Training data

Built from the canonical corpus with a corrected adapter, as a local candidate: three sources of six, because xlam and button are gated upstream and LoopTool's source was not located.

source	trainable	with tool calls
glaive-function-calling-v2	97,112	48,723 (50.2%)
when2call	4,000	0
toolace	673	0

The Glaive records are the contribution. The published pipeline could not parse this revision's call format and silently dropped them — 51,034 records rejected as orphaned tool results. Two deviations had to be handled: the blocks are never closed by </functioncall> (0 of 67,481 have one), and arguments is a single-quoted Python-style string holding JSON with lower-case booleans, valid as neither JSON nor a Python literal. The corrected adapter reads 98.5% of those turns. 154,760 records were quarantined rather than repaired, and no tool schema was coerced into validity.

This corpus has not had the semantic contamination review that the v1 corpus received, and it is not claimed to have. Its evaluation-side sources are excluded by the same policy as v1.

Intended use and limits

Intended for research on tool-calling decision boundaries. Not evaluated for safety, not aligned, and not intended for autonomous tool use. Expect: over-clarification on ambiguous requests, degraded behaviour outside the training distribution, and no reasoning or safety tuning.