formcheck-vlm prototype

A 41.2M-parameter VLM (vision encoder + char-level decoder, trained from scratch) that extracts checkbox / radio / fillbox states from form images and answers with prompt-specified JSON.

Results (48 held-out synthetic val samples, after 4000 steps on 30k pages)

metric	value
JSON parse rate	0.48
per-control state accuracy (label+state both correct)	0.027
full-output exact match	0.00
teacher-forced next-char accuracy	0.91

The pipeline (synthetic data generation, training, evaluation, generation) is verified end-to-end; the model emits correctly-schemed JSON about half the time but does not yet read printed labels/states accurately — expected for a 41M model trained ~2 epochs. Known levers for the next iteration: longer training / more data, higher effective image resolution for small controls, state-class rebalancing (the "empty" prior dominates), and per-control decoding.

Spec

• States: empty, checked_tick, checked_cross, checked_mark, filled, filled_partial, strikethrough, selected
• Control types: checkbox, radio, fillbox
• Prompt templates: 6 (list-all, label→state map, marked-only, empty-only, single lookup, group-by-type)
• Architecture: ViT (8 layers, dim 384, patch 16, 384px input → 576 tokens) + 8-layer char-level decoder (dim 512), prefix-LM attention, greedy generation
• Data: fully synthetic rendered forms with exact JSON ground truth; 40% digital / 40% scan-simulated / 20% photo-simulated
• Config: {"vocab_size": 131, "img_size": 384, "patch": 16, "decoder_dim": 512, "decoder_depth": 8, "decoder_heads": 8, "vis_dim": 384, "vis_depth": 8, "vis_heads": 6, "params_m": 41.2}

Files

• pytorch_model.bin — weights (torch state_dict)
• config.json — architecture config
• job_train.py — self-contained training script (data generation + training + eval + push)

Training

N_TRAIN=30000 N_VAL=500 STEPS=4000 BS=16 LR=3e-4 python job_train.py

Trained on a10g-small for 33 minutes ($0.60).