SuhaoYu1020/semantic-image-arith-ckpts

semantic-image — arithmetic image-edit checkpoints

22 checkpoints from the experiments in semantic-image, selected because each one is a number in the results table — not because it is the best model. Several are here precisely because they fail.

The task: given a 512×512 image of Q: 21 + 60 = ? / A:, edit it so the answer appears after A:. Input and target differ only in the answer glyphs, so the pixel loss isolates arithmetic. The metric is Qwen3-VL-2B OCR accuracy on a 1,000-example held-out test set (the older nearest-neighbour render_acc proxy overstates it and is not used here).

What is here

Each folder is <family>/<run>/ and holds transformer/, meta.json, train_args.json (the exact argv) and CARD.md.

noemit-2x2 — does the DiT compute, or does the text encoder?

Pure MSE, no emitted token; the answer must be computed inside the network.

checkpoint	test OCR
M4 train-TE, scratch-2B DiT	87.7%	the winner
M3 freeze-TE, scratch-2B DiT	33.2%	the DiT alone still does about a third
TE frozen at M4's trained weights, scratch-2B	87.6%	so the TE must be good, not training
M4 train-TE, pretrained-20B DiT	0.9%	pretrained DiT takes the render shortcut
M3 freeze-TE, pretrained-20B DiT	1.0%

Zero train/test gap on the winner (88.0% train vs 87.7% test) — this is generalization, not memorization.

seeds — how much of that is seed noise

M4 across seeds 43/44/45: 35.8% / 89.9% / 88.8%. M3: 25.8% / 7.0% / 14.8%. Seed 43 for M4 is a genuine outlier, kept here so the variance is not hidden.

emit-budget — answer tokens as a compute budget

The text encoder emits K tokens that are re-encoded and fed to the DiT. K is a compute budget, and the response is monotone:

K	scratch-2B	pretrained-20B
1	44.1%	0.8%
2	74.6%	15.6%
4	88.8%	87.2%

The pretrained DiT collapses at small K (it ignores a partially-hinted answer and falls back to rendering), and only catches up once the hint is nearly complete.

emit-slots — the same axis at 300 epochs

1 slot / answers ≤9, scratch: 99.2%. 2 slots / ≤99, pretrained: 89.2%. 3 slots / 0–198, pretrained: 69.7%.

t2i, holdout

The text-to-image port (no input image) reaches only 10.1% / 5.4% — the edit setting is doing real work. holdout is the train≠test split with the answer carried by the DiT + text decoder: 88.7%.

Missing on purpose

Five checkpoints that produced published numbers no longer exist: those runs continued past the evaluated step and --keep_last_ckpts pruned the older ones (including one explicitly named step_45000_ep10_KEEP). The affected numbers — emit3slots scratch 90.6% and 91.1%, emit3slots pretrained 60.0%, emit2slots scratch 92.1% — are reproducible from the launchers but not restorable as weights. Their eval reports and per-row OCR output survive in the dataset repo.

Loading

from diffusers import QwenImageEditPipeline
from diffusers.models import QwenImageTransformer2DModel

tr = QwenImageTransformer2DModel.from_pretrained(
        "SuhaoYu1020/semantic-image-arith-ckpts",
        subfolder="noemit-2x2/<run>/transformer", torch_dtype="bfloat16")
pipe = QwenImageEditPipeline.from_pretrained("Qwen/Qwen-Image-Edit", transformer=tr)

Weights only — no optimizer state. freeze_te runs ship the DiT alone (3.8 GB scratch / ~38 GB pretrained); pair them with the stock Qwen-Image-Edit text encoder. Runs that trained the text encoder include it (19 GB / ~53 GB).

diffusers must be the pinned dev build, git+https://github.com/huggingface/diffusers@6abf75263a09a3e7a62458f544ce2fac28568fe2.

Citation / code

Code, launchers and the full results table: the semantic-image repository. Datasets: SuhaoYu1020/semantic-image-arith.