dCoder30/gamedoctor-qwen38-adapters

GameDoctor agentic-SFT adapters (Qwen3.8-27B)

Six LoRA adapters from a three-stage agentic-SFT ablation: distilling a stronger teacher's browser-game-building skill into Qwen3.8-27B, then measuring whether each added data stage helps.

All adapters share one base model and one LoRA config. They differ only in the training corpus.

The adapters

folder	stage	training corpus	rows
stage1-gen	1	generation trajectories only	409
stage2-gen-plan-v1	2	gen + plan	863
stage2-gen-plan-reweighted	2	gen + plan, plan downsampled to 25%	525
stage3-gen-plan-repair-v1	3	gen + plan + repair	1,008
stage3-gen-plan-repair-reweighted	3	as above, plan+repair at 25%	572
stage3-reweighted-alt	3	identical config, separate run (redundancy)	572

"v1" vs "reweighted" use the same examples. Reweighting changes only how often each kind is seen. In stage 2 that moves generation from 47.4% to 77.9% of training steps. The motivation was that stages 2 and 3 showed no gain over stage 1, and generation appeared to be outvoted.

Two other suspected data defects — leaked absolute file paths, and plan targets written in past tense — were investigated and disproven, so neither was changed. Reweighting is the only difference.

Architecture note (important for merging)

Qwen3.8-27B is a hybrid: 48 linear-attention (Mamba-style) blocks, 16 full-attention blocks, 64 MLP blocks — 496 LoRA target modules.

Training nests the decoder under model.language_model.* while serving flattens it to model.*. A merge that does not rewrite both the tensor keys and the target_modules regex silently matches 0 of 496 modules and produces a model that loads fine and contains none of the fine-tuning. Verify after merging:

target modules after remap: 496 (expect 496)
still containing language_model: 0 (expect 0)
keys base=1199 merged=1199 identical=True
chat_template chars: base=8952 merged=8952 match=True

The full tensor count is 1,199 = 992 decoder + 333 vision + 15 MTP. A naive merge drops the 348 vision/MTP tensors; repair_merge.py in the code repo lifts them back and refuses if any missing key is under model.language_model.*.

Training configuration

Identical across all six runs, so stages remain comparable:

• ms-swift 4.5.2 LoRA SFT, QLoRA bnb 4-bit nf4 + double quant
• Ulysses sequence parallelism, SP=4
• 1 epoch, batch size 1, no gradient accumulation (too few updates at this data size), loss unmodified
• assistant turns only are supervised (ms-swift messages default)
• 4×H100-80GB per run

What reweighting did to training

signal	v1	reweighted
gradient clipping rate (S2)	64.2%	32.9%
gradient clipping rate (S3)	66.0%	37.3%
grad-norm, median	1.576	0.328
token accuracy	0.848	0.944

Every training-time signal improved. None of them predicted whether the resulting game would run — see the eval repo.

Results, in brief

On three held-out briefs with zero overlap with any training set, scored by a BUILD gate (a headless browser must load dist/index.html) plus a vision-model rubric:

model	mean reward	games that run
untuned base	0.077	1 / 2
stage-3 v1	0.293	3 / 3
stage-2 reweighted	0.167	1 / 2
stage-3 reweighted, run 1	0.000	0 / 2
stage-3 reweighted, run 2	0.149	1 / 2

Read the last two rows before ranking anything. They are the same adapter evaluated twice, and they differ by more than the gap between any two different models. At two comparable briefs, this evaluation is noisier than the effect it was built to measure. Treat the ordering as unresolved.

The recurring failure is a self-inflicted one: the reweighted adapters tend to register a single Phaser scene class under many level keys while that class hardcodes its own key, so the second registration collides and the game dies on load. It appeared in 3 of 5 games from reweighted adapters and in none from the v1 or base models — but it is a frequent habit, not deterministic.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.8-27B", trust_remote_code=True)
model = PeftModel.from_pretrained(base, "<this-repo>/stage3-gen-plan-repair-v1")

To serve with vLLM, merge first and verify the counts above; runtime LoRA was not used for any measurement here and would not be comparable.

Limitations

• Two comparable briefs per arm. Directional only.
• stage2-gen-plan-v1 was never evaluated on the uncontaminated briefs, so the stage-2 v1-vs-reweighted comparison is missing.
• One reported score rests on a single judged pass; the other two judge calls returned empty and were excluded rather than scored zero.

---

Pipeline code, reports and scores: WenyiWU0111/OpenGame-reproduce · gamedoctor_sft/

Companion repos: adapters · training data · eval results