OpenGrad — Qwen3.5-2B, M1 DPO (negative result)

Direct Preference Optimization of Qwen/Qwen3.5-2B on the When2Call preference split, produced by OpenGrad.

This checkpoint is published as a negative result and is not recommended for use. It is kept because the way it fails is the informative part.

Known limitation: general-capability regression

This is not the promoted model, and it should not be used as a general-purpose assistant. Its run was rejected. The checkpoint the project promoted is a different run, OpenGrad-Qwen3.5-2B-M1-DPO-CanonicalV2-Final-v2 @30, whose card carries its own caveats.

This checkpoint (step 300) is DPO applied directly to the base model: no SFT parent (parent_experiment_id: null), the reference is the initial policy (reference: initial_policy), and the preference data is when2call_pref_v1. It shows a general-capability regression associated with tool-policy post-training on When2Call-derived data. It appears after SFT (M0) and also after DPO applied directly to the base (this model), so it is not specific to SFT. Causation is not established: one lineage, one seed, no replicate.

Measured in the H200 capability campaign (vLLM 0.29.0, greedy decoding), against the base model:

benchmark	Qwen3.5-2B base	this checkpoint (step 300)
GSM8K 0-shot accuracy	67.4%	17.3%
GSM8K 0-shot refusals	0 of 1,319	933 of 1,319 (70.7%)
GSM8K 8-shot accuracy	70.4%	70.4%
IFEval prompt-level strict	67.8%	43.4%
MMLU-Pro	49.0%	unmeasured

With eight exemplars it answers as many GSM8K questions correctly as the base (929 of 1,319 each); asked with no exemplars, it declines 70.7% of them. Its MMLU-Pro was not measured at the corrected 2,048-token budget: that run was stopped for budget (TERMINATED_BUDGET) at about 39% of prompts, and no results were retained. The earlier 768-token run was superseded after it truncated 49.7% of items. Sources: results/final_campaign_verdict.json and results/benchmarks/h200/capability_v1/M1_DPO_HISTORICAL/*_scores.json in the repository.

Its only repeat, qwen35_2b_m1_dpo_v1_restore (below), is recorded as NON_REPRODUCIBLE_REPEAT: its trajectory did not reproduce this run's.

Checkpoint availability, and a correction

Read this before the numbers below.

This run measured three checkpoints — steps 100, 200, and 300 — but only step 300 is in this repository. The weights for steps 100 and 200 were deleted from local storage after the run and had never been uploaded, so they no longer exist anywhere and cannot be recovered. The deleted step 100 was the best of the three (call_f1 0.1715 against 0.0258 at step 300), so the checkpoint that is published is the weakest state the run reached, while the table below reports all three.

The run was then repeated to regenerate them, with the base checkpoint, preference file (SHA-256 474a8bb1…), hyperparameters, seed, and software environment all verified identical. It did not reproduce the numbers.

step	`call_f1` original	`call_f1` repeat
100	0.1715	0.1036
200	0.0419	0.1186
300	0.0258	0.0153

The original declined monotonically with step 100 best; the repeat peaked at step 200. The regenerated weights were not uploaded, because they are a different model and publishing them under these step labels would attach one run's measurements to the other's. Consequences, stated plainly:

The per-step numbers below are correct for the weights this run produced. They were re-derived from this run's saved per-example predictions and match to within metrics.json rounding (max difference 4.9e-07, across 36 metrics), so the scoring can still be checked even though the models cannot be re-run.
The claim that degradation is monotone from the first measured checkpoint, and that step 100 is therefore the best stopping point, is withdrawn as a single-run observation. One run never supported a statement about the shape of the curve.
The direction of the finding is unchanged. Both runs eliminate over-calling, collapse call recall, and raise precision.

Full account, including what happened and what changed as a result: INC-0001 in the OpenGrad incident log. This project publishes its operational mistakes alongside its results rather than presenting the surviving artefact as if it were the intended one.

Why it is here

The hypothesis was that DPO on preference pairs would fix the baseline's miscalibrated call/no-call boundary. B0 calls a tool on 64.3% of examples whose correct answer is not a call while recalling 97.2% of gold CALLs — it over-calls badly. The preference data was chosen to address exactly that: 1,741 pairs in which the preferred response is the correct decision, not always a tool call. All four behaviours appear as chosen in near-equal numbers, and both TOOLCALL → CLARIFY (801) and CLARIFY → TOOLCALL (1,018) directions are present.

DPO did what it was asked, and then kept going.

step	call_f1	precision	recall	over-call	clar_ok	unsup_ok	weights
100	0.1715	0.7669	0.0965	0.0161	0.9179	0.6162	not available
200	0.0419	0.6829	0.0216	0.0055	0.8726	0.6317	not available
300	0.0258	0.6800	0.0131	0.0034	0.9623	0.4571	in this repo
B0	0.6191	0.4542	0.9722	0.6425	0.1009	0.0131	—

Over-calling was eliminated (0.6425 → 0.0034), precision rose 0.454 → 0.680, clarification accuracy 0.101 → 0.962, and unsupported accuracy 0.013 → 0.457. Call recall fell 0.9722 → 0.0131. The model stopped calling tools. In this run the decline was monotone from the first measured checkpoint, so there was never a point at which it beat doing nothing — but see the correction above: a repeat of the run did not reproduce that trajectory.

Why it failed

Two reasons, and the second is the interesting one.

Over-optimisation. The implicit-reward margin grows 0 → 23.4 with preference accuracy pinned at 1.000, on 1,741 pairs over 300 steps. With beta = 0.1 a margin of 23 is enormous. The preference set is small for this objective.

The starting point. B0 is not a balanced model. It recalls 97.2% of gold CALLs and 1.3% of gold UNSUPPORTEDs, so every signal that moves it off always-call reduces call recall. Per-example balance in the preference data cannot prevent that, because the model has far more "stop calling" to learn than "keep calling". The same directional failure appears under supervised fine-tuning for the same reason.

An earlier version of this card added a third reason: the companion M0 corpus carried tool-call supervision for only 9 of 55,719 trainable records, and DPO "cannot supply" a behaviour nearly absent from its training signal. That does not apply to this run. It is DPO applied directly to the base model, which already produces tool calls (97.2% CALL recall), on when2call_pref_v1 rather than that corpus. The two reasons above are the ones the evidence supports.

Training


Base model	`Qwen/Qwen3.5-2B` @ `15852e8c16360a2fea060d615a32b45270f8a8fc`
Reference	frozen copy of the initial policy
beta	0.1
LR	5e-6, 20-step warmup
Steps	300
Batch	2 x 2 accumulation
Window	2,048 tokens
Pairs	1,741 (of 9,000; the rest have tool schemas using Python type hints where JSON Schema is required)
Hardware	1x A100-SXM4-80GB, 10.1 min, 29.5 GiB peak

Loss is computed over completion tokens only, with the causal shift applied to the mask, and the reference model is frozen and never receives gradients. Run through opengrad train; the preference file's SHA-256 is recorded in the run.

Reproducing the numbers

The three per-checkpoint evaluations are recorded in the OpenGrad repository under runs/qwen35_2b_m1_dpo_v1/eval/. The recorded metrics are the outcome of scoring those predictions, so they can be re-derived rather than taken on trust:

from opengrad.evaluation.routing import routing_metrics
# actual = [row["expected_decision"] for row in predictions]
# predicted = [row["prediction"]["decision"] for row in predictions]

Reproducing the models is not possible: two of the three no longer exist, and the repeat run did not match.

Intended use and limits

Research artefact. Not a general-purpose assistant (see the capability regression above), not the promoted model, not safety-tuned, not aligned, and not intended for autonomous tool use. Given the measured metrics, it is expected to under-call tools severely and should not be deployed for tasks requiring tool invocation.