dCoder30/evogame-qwen38-adapters

EvoGame agentic-SFT adapters (Qwen3.8-27B)

Six LoRA adapters from a three-stage agentic-SFT ablation: distilling a stronger teacher's browser-game-building skill into Qwen3.8-27B, then measuring whether each added data stage helps.

All adapters share one base model and one LoRA config. They differ only in the training corpus.

The adapters

folder	stage	training corpus	rows
stage1-gen	1	generation trajectories only	409
stage2-gen-plan-v1	2	gen + plan	863
stage2-gen-plan-reweighted	2	gen + plan, plan downsampled to 25%	525
stage3-gen-plan-repair-v1	3	gen + plan + repair	1,008
stage3-gen-plan-repair-reweighted	3	as above, plan+repair at 25%	572
stage3-reweighted-alt	3	identical config, separate run (redundancy)	572

"v1" vs "reweighted" use the same examples. Reweighting changes only how often each kind is seen. In stage 2 that moves generation from 47.4% to 77.9% of training steps. The motivation was that stages 2 and 3 showed no gain over stage 1, and generation appeared to be outvoted.

Two other suspected data defects — leaked absolute file paths, and plan targets written in past tense — were investigated and disproven, so neither was changed. Reweighting is the only difference.

Round 2 (v2, both engines)

Round 1 trained on web games only. Round 2 retrains every stage on corpora that carry both engines (web + Godot), so a stage's gain — if any — has to hold on an engine the round-1 arms never saw.

Folder	Stage	Corpus	Steps	max_length
v2/s1-split	1	s1_split_train.jsonl	3,125	33,792
v2/s1-baseline	1	s1_train_lf.jsonl	1,294	57,344
v2/s2-gen-plan	2	s2_both_engines.jsonl	1,602	57,344
v2/s3-gen-plan-repair	3	s3_both_engines.jsonl	2,111	57,344

S1 split vs baseline is the same generation data cut differently: baseline trains on whole trajectories, split on sliding windows at a shorter max_length — ~2.4× the steps at ~3× lower seconds/step.

Checkpoints shipped here: checkpoint-3100 (s1-split, the last save before the 3,125-step end), checkpoint-1294, checkpoint-1602, checkpoint-2111. Every adapter_model.safetensors is 467,062,560 bytes; SHA256SUMS sits next to each.

Measured cost

Run	s/step	Peak mem	Wall span	Loss (last 50)	Token acc
s1-split	29.1	50.3 GiB	34.1 h	0.302	0.907
s1-baseline	90.9	74.2 GiB	37.3 h	0.253	0.899
s2-gen-plan	86.2	74.0 GiB	38.4 h	0.315	0.858
s3-gen-plan-repair	63.9	74.3 GiB	38.6 h	0.425	0.941

Seconds/step and peak memory come from the last training segment only — train.log is rewritten on every resume. Wall span is first to last checkpoint save and includes crash, queue and resume time. The summary train_loss that ms-swift reports covers only the resumed segment, so the loss column is the mean of the last 50 per-step values from trainer_state.json instead.

cuDNN SDPA crashes (all four runs hit this)

S1 baseline (step 1070), S2 (steps 109 and 1486) and S3 (steps 521 and 1775) all died inside torch SDPA with:

RuntimeError: Expected mha_graph->execute(handle, variant_pack, workspace_ptr.get()).is_good() to be true

It is a cuDNN fused-attention kernel failure triggered by one row's shape, so it is deterministic and a plain resume replays it. TORCH_CUDNN_SDPA_ENABLED=0 is ignored by this torch build. What works is disabling the backend in code, in every worker — a sitecustomize.py on the training PYTHONPATH:

import torch
torch.backends.cuda.enable_cudnn_sdp(False)

That leaves the flash / mem-efficient / math kernels, which do not go through mha_graph. Memory is unchanged (~74 GiB); throughput costs ~21% (88.7 → 73 s/it measured on S2). With it set, S2 cleared the row that had killed it twice.

Stage 3's teacher differs

Most stage-3 repair rows were produced with z-ai/glm-5.3-flash as the repair model, so S3 distils a different teacher than S1 and S2. Any S3-vs-S2 difference confounds "repair data" with "different teacher".

How these adapters score

Four held-out briefs — two web, two Godot — none of which appear in any training set, judged by openai/gpt-5.5 against per-task rubrics. Web cells are the mean of three judged passes; Godot cells are one rubric pass.

Adapter	platformer	topdown	metroidvania	rhythm	mean
v2/s1-split	0.407	0.422	0.524	0.589	0.486
v2/s1-baseline	0.406	0.475	0.511	0.509	0.475
v2/s2-gen-plan	0.030	0.528	0.439	0.396	0.348
v2/s3-gen-plan-repair	0.000	0.356	0.404	0.541	0.325

The two stage-1 adapters are tied (0.486 vs 0.475 is inside this benchmark's noise, and their platformer scores match to three decimals), and neither later stage improves on stage 1. Both later stages are pulled down by one dead game rather than uniform weakness: S2's platformer renders an empty canvas after its level scene starts, and S3's fails the build gate outright (this.gateZone.body.setAllowGravity is not a function). At two briefs per engine a single dead game moves a mean by ~0.1, so treat the ordering as unresolved rather than as a ranking.

Replays for these scores were captured on dedicated CPUs. An earlier round measured on a throttled 8-CPU box scored the same games far lower (S2's metroidvania 0.114 vs 0.439 here) because recordings ended before the games drew anything; those numbers were discarded.

Full protocol, per-requirement breakdowns, generated games and the recordings the judge saw: dCoder30/evogame-eval.

Architecture note (important for merging)

Qwen3.8-27B is a hybrid: 48 linear-attention (Mamba-style) blocks, 16 full-attention blocks, 64 MLP blocks — 496 LoRA target modules.

Training nests the decoder under model.language_model.* while serving flattens it to model.*. A merge that does not rewrite both the tensor keys and the target_modules regex silently matches 0 of 496 modules and produces a model that loads fine and contains none of the fine-tuning. Verify after merging:

target modules after remap: 496 (expect 496)
still containing language_model: 0 (expect 0)
keys base=1199 merged=1199 identical=True
chat_template chars: base=8952 merged=8952 match=True

The full tensor count is 1,199 = 992 decoder + 333 vision + 15 MTP. A naive merge drops the 348 vision/MTP tensors; repair_merge.py in evogame-sft-pipeline/ lifts them back and refuses if any missing key is under model.language_model.*.

Training configuration

Identical across all six runs, so stages remain comparable:

• ms-swift 4.5.2 LoRA SFT, QLoRA bnb 4-bit nf4 + double quant
• Ulysses sequence parallelism, SP=4
• 1 epoch, batch size 1, no gradient accumulation (too few updates at this data size), loss unmodified
• assistant turns only are supervised (ms-swift messages default)
• 4×H100-80GB per run

What reweighting did to training

signal	v1	reweighted
gradient clipping rate (S2)	64.2%	32.9%
gradient clipping rate (S3)	66.0%	37.3%
grad-norm, median	1.576	0.328
token accuracy	0.848	0.944

Every training-time signal improved. None of them predicted whether the resulting game would run — see the eval repo.

Results, in brief

On three held-out briefs with zero overlap with any training set, scored by a BUILD gate (a headless browser must load dist/index.html) plus a vision-model rubric:

model	mean reward	games that run
untuned base	0.077	1 / 2
stage-3 v1	0.293	3 / 3
stage-2 reweighted	0.167	1 / 2
stage-3 reweighted, run 1	0.000	0 / 2
stage-3 reweighted, run 2	0.149	1 / 2

Read the last two rows before ranking anything. They are the same adapter evaluated twice, and they differ by more than the gap between any two different models. At two comparable briefs, this evaluation is noisier than the effect it was built to measure. Treat the ordering as unresolved.

The recurring failure is a self-inflicted one: the reweighted adapters tend to register a single Phaser scene class under many level keys while that class hardcodes its own key, so the second registration collides and the game dies on load. It appeared in 3 of 5 games from reweighted adapters and in none from the v1 or base models — but it is a frequent habit, not deterministic.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.8-27B", trust_remote_code=True)
model = PeftModel.from_pretrained(base, "<this-repo>/stage3-gen-plan-repair-v1")

To serve with vLLM, merge first and verify the counts above; runtime LoRA was not used for any measurement here and would not be comparable.

Limitations

• Two comparable briefs per arm. Directional only.
• stage2-gen-plan-v1 was never evaluated on the uncontaminated briefs, so the stage-2 v1-vs-reweighted comparison is missing.
• One reported score rests on a single judged pass; the other two judge calls returned empty and were excluded rather than scored zero.

---

Pipeline code, reports and scores: WenyiWU0111/OpenGame-reproduce · evogame-sft-pipeline/ · evogame-data-pipeline/

Companion repos: adapters · training data · eval results