kaktus-bbox-lora-v3

LoRA adapter for Qwen/Qwen3-VL-4B-Instruct. Input: a rendered document page. Output: one JSON object with the page's layout elements and bounding boxes.

Successor to kaktus-bbox-lora-v1, trained on a cleaned version of the same dataset. On OmniDocBench (English) it improves detection F1 from 0.473 to 0.570 and mean IoU from 0.603 to 0.691.

{"elements": [
  {"type": "section_header", "bbox": [48, 48, 307, 68], "text": "1 Introduction"},
  {"type": "paragraph", "bbox": [48, 78, 948, 112], "text": "..."},
  {"type": "table", "bbox": [48, 300, 948, 610], "text": "..."}
]}

Format:

• bbox is [x0, y0, x1, y1], integers on a 0-1000 grid, origin top-left.
• 16 element types: title, section_header, paragraph, list, table, figure, chart, caption, page_header, page_footer, page_number, form_field, equation, signature, stamp, other.
• A blank page returns {"elements": []}.

Training data

10,000 PDFs sampled from FinePDFs (Common Crawl), rendered with pdfboss at about 1224 px, labeled by kimi-k2.6, then cleaned:

• 268,971 boxes snapped to the ink extent of their content (tight boxes instead of loose ones).
• 24,114 empty boxes removed, 2,193 degenerate boxes removed, 2,805 off-vocabulary types remapped, all coordinates clipped to the 0-1000 grid.
• 4,316 under-labeled pages re-extracted by the teacher; the better answer per page was kept, decided by ink-coverage measurement on the rendered page (1,022 pages upgraded).
• Label quality was checked against DocLayout-YOLO as an independent detector: agreement rose on both axes after cleaning (0.602 to 0.630 and 0.475 to 0.523).

Final: 60,315 page/JSON pairs, doc-disjoint eval split.

Training

Base	Qwen/Qwen3-VL-4B-Instruct, ViT and aligner frozen
Method	LoRA r=8, alpha=32, all-linear, lr 1e-4, bf16
Schedule	1 epoch, 3,770 steps, effective batch 16 (8x H100, DDP)
Context	max_length 6144, IMAGE_MAX_TOKEN_NUM 1024
Framework	ms-swift 4.5.2

Evaluation

OmniDocBench (human-annotated, category-mapped, greedy IoU matching):

English subset (755 pages)	v1	v3
Detection F1 at IoU >= 0.5	0.473	0.570
Precision / Recall	0.539 / 0.421	0.693 / 0.485
Mean IoU of matched boxes	0.603	0.691
Class accuracy on matches	90.6%	91.2%
Valid JSON	95.4%	90.2%

Internal held-out eval (1,652 pages, cleaned labels): F1 0.646 at IoU >= 0.5, mean IoU 0.710, type accuracy 93.3%, zero out-of-range boxes.

Usage

Tested with ms-swift:

IMAGE_MAX_TOKEN_NUM=1024 swift infer \
  --adapters 4thel00z/kaktus-bbox-lora-v3 \
  --infer_backend pt --max_new_tokens 8192 --temperature 0

With transformers + peft:

from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel

base = "Qwen/Qwen3-VL-4B-Instruct"
model = AutoModelForImageTextToText.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "4thel00z/kaktus-bbox-lora-v3")
processor = AutoProcessor.from_pretrained(base)

Prompt with the page as the image and the instruction "Extract this page. Respond with only the JSON object." The system prompt is in args.json. Use max_new_tokens >= 8192 for dense pages and clip bbox values to [0, 1000].

Limitations

• Training labels are model-generated (kimi-k2.6, cleaned as described above), not human annotations.
• The corpus is English (FinePDFs eng_Latn). On non-English pages JSON validity drops (72.1% on the full OmniDocBench mix including Chinese).
• Pages that need more than the generation budget truncate mid-JSON; use 8192 tokens or more.

Training history

TRAINING_HISTORY.md records the data cleaning steps, hardware, and the exact data snapshot. Training arguments are in args.json.