arrochi112/OpenGrad-Qwen3.5-2B-M1-DPO

OpenGrad — Qwen3.5-2B, M1 DPO (negative result)

Direct Preference Optimization of Qwen/Qwen3.5-2B on the When2Call preference split, produced by OpenGrad.

This checkpoint is published as a negative result and is not recommended for use. It is kept because the way it fails is the informative part.

Checkpoint availability, and a correction

Read this before the numbers below.

This run measured three checkpoints — steps 100, 200, and 300 — but only step 300 is in this repository. The weights for steps 100 and 200 were deleted from local storage after the run and had never been uploaded, so they no longer exist anywhere and cannot be recovered. The deleted step 100 was the best of the three (call_f1 0.1715 against 0.0258 at step 300), so the checkpoint that is published is the weakest state the run reached, while the table below reports all three.

The run was then repeated to regenerate them, with the base checkpoint, preference file (SHA-256 474a8bb1…), hyperparameters, seed, and software environment all verified identical. It did not reproduce the numbers.

step	call_f1 original	call_f1 repeat
100	0.1715	0.1036
200	0.0419	0.1186
300	0.0258	0.0153

The original declined monotonically with step 100 best; the repeat peaked at step 200. The regenerated weights were not uploaded, because they are a different model and publishing them under these step labels would attach one run's measurements to the other's. Consequences, stated plainly:

• The per-step numbers below are correct for the weights this run produced. They were re-derived from this run's saved per-example predictions and match to within metrics.json rounding (max difference 4.9e-07, across 36 metrics), so the scoring can still be checked even though the models cannot be re-run.
• The claim that degradation is monotone from the first measured checkpoint, and that step 100 is therefore the best stopping point, is withdrawn as a single-run observation. One run never supported a statement about the shape of the curve.
• The direction of the finding is unchanged. Both runs eliminate over-calling, collapse call recall, and raise precision.

Full account, including what happened and what changed as a result: INC-0001 in the OpenGrad incident log. This project publishes its operational mistakes alongside its results rather than presenting the surviving artefact as if it were the intended one.

Why it is here

The hypothesis was that DPO on preference pairs would fix the baseline's miscalibrated call/no-call boundary. B0 calls a tool on 64.3% of examples whose correct answer is not a call while recalling 97.2% of gold CALLs — it over-calls badly. The preference data was chosen to address exactly that: 1,741 pairs in which the preferred response is the correct decision, not always a tool call. All four behaviours appear as chosen in near-equal numbers, and both TOOLCALL → CLARIFY (801) and CLARIFY → TOOLCALL (1,018) directions are present.

DPO did what it was asked, and then kept going.

step	call_f1	precision	recall	over-call	clar_ok	unsup_ok	weights
100	0.1715	0.7669	0.0965	0.0161	0.9179	0.6162	not available
200	0.0419	0.6829	0.0216	0.0055	0.8726	0.6317	not available
300	0.0258	0.6800	0.0131	0.0034	0.9623	0.4571	in this repo
B0	0.6191	0.4542	0.9722	0.6425	0.1009	0.0131	—

Over-calling was eliminated (0.6425 → 0.0034), precision rose 0.454 → 0.680, clarification accuracy 0.101 → 0.962, and unsupported accuracy 0.013 → 0.457. Call recall fell 0.9722 → 0.0131. The model stopped calling tools. In this run the decline was monotone from the first measured checkpoint, so there was never a point at which it beat doing nothing — but see the correction above: a repeat of the run did not reproduce that trajectory.

Why it failed

Two reasons, and the second is the interesting one.

Over-optimisation. The implicit-reward margin grows 0 → 23.4 with preference accuracy pinned at 1.000, on 1,741 pairs over 300 steps. With beta = 0.1 a margin of 23 is enormous. The preference set is small for this objective.

The starting point. B0 is not a balanced model. It recalls 97.2% of gold CALLs and 1.3% of gold UNSUPPORTEDs, so every signal that moves it off always-call reduces call recall. Per-example balance in the preference data cannot prevent that, because the model has far more "stop calling" to learn than "keep calling". The same directional failure appears under supervised fine-tuning for the same reason.

The corpus used in the companion M0 experiment contains tool-call supervision for 9 of 55,719 trainable records (0.0162%) — the records carrying tool calls were the ones being dropped by a parsing defect in the published pipeline. DPO reweights behaviours a policy can already produce; it cannot supply one that is nearly absent from the training signal. Fixing the corpus is what moved the metric (see arrochi112/OpenGrad-Qwen3.5-2B-M0-SFT-CorpusV2).

Training

Base model	Qwen/Qwen3.5-2B @ 15852e8c16360a2fea060d615a32b45270f8a8fc
Reference	frozen copy of the initial policy
beta	0.1
LR	5e-6, 20-step warmup
Steps	300
Batch	2 x 2 accumulation
Window	2,048 tokens
Pairs	1,741 (of 9,000; the rest have tool schemas using Python type hints where JSON Schema is required)
Hardware	1x A100-SXM4-80GB, 10.1 min, 29.5 GiB peak

Loss is computed over completion tokens only, with the causal shift applied to the mask, and the reference model is frozen and never receives gradients. Run through opengrad train; the preference file's SHA-256 is recorded in the run.

Reproducing the numbers

The three per-checkpoint evaluations are recorded in the OpenGrad repository under runs/qwen35_2b_m1_dpo_v1/eval/. The recorded metrics are the outcome of scoring those predictions, so they can be re-derived rather than taken on trust:

from opengrad.evaluation.routing import routing_metrics
# actual = [row["expected_decision"] for row in predictions]
# predicted = [row["prediction"]["decision"] for row in predictions]

Reproducing the models is not possible: two of the three no longer exist, and the repeat run did not match.

Intended use and limits

Research artefact. Not safety-tuned, not aligned, and not intended for autonomous tool use. Given the measured metrics, it is expected to under-call tools severely and should not be deployed for tasks requiring tool invocation.