amad-vlm6

amad-vlm6 is a TIES merge of two Arabic OCR fine-tunes of Qwen2.5-VL-7B-Instruct: amad-iq/amad-vlm5 and AhmedZaky1/DIMI-Arabic-OCR-V2.

📄 Full technical report — method, per-dataset results, and an error-type decomposition showing what the merge actually fixed.

Results

KITAB-Bench ocr-eval, 13 datasets, 3,760 images, final-answer scoring with the benchmark's unmodified metrics.

	amad-vlm5	amad-vlm6
CER ↓ (macro)	0.254	0.235
CHrF ↑ (macro)	81.05	81.68
CER ↓ (10 leak-free datasets)	0.321	0.236
CHrF ↑ (10 leak-free datasets)	77.35	82.46

Largest per-dataset gains: khattparagraph CER 0.711 → 0.088, evarest 0.291 → 0.142, hindawi 0.241 → 0.137.

What the gain is. Error-type decomposition (see the report) shows the improvement is behavioral: the merge cuts amad-vlm5's thinking rate on page-level inputs from 45–98% to 5–18% and removes its truncated and empty outputs, so insertions and deletions collapse. Reading accuracy itself (substitution rate) is worse on 10 of 13 datasets and better only on adab and evarest. Hallucination is not cancelled but moved: on isippt and muharaf the merged model over-generates where amad-vlm5 did not. Four per-dataset wins and three losses are statistically significant (bootstrap); five are not.

Files

Repository	Format	Use
this repo	bf16 safetensors	Transformers, vLLM, further fine-tuning
amad-vlm6-GGUF	Q8_0 + mmproj	llama.cpp, LM Studio

Usage

import re, torch
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration

repo = "amad-iq/amad-vlm6"
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    repo, dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(repo)

messages = [{"role": "user", "content": [
    {"type": "image", "image": "page.png"},
    {"type": "text", "text": "Extract the text in the image. Give me the final text, nothing else."}]}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt").to(model.device)

out = model.generate(**inputs, max_new_tokens=4096, do_sample=False, repetition_penalty=1.05)
raw = processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0]
print(re.sub(r"<think>.*?</think>", "", raw, flags=re.S).strip())

The model inherits a thinking behaviour from one parent: on dense pages it may emit <think>…</think> before the transcription. Keep only the text after the last </think>, and allow 4,096 new tokens — a 500-token budget truncates dense pages.

Limitations

• Reading regresses on 10 of 13 datasets (substitution rate); the CER wins come from output discipline, not perception. Macro WER is worse (0.361 → 0.463), driven by three datasets with catastrophic rows (historyar, muharaf, isippt), not by a general segmentation problem.
• Two large regressions from imported over-generation: isippt CER 0.054 → 0.541 (insertions 0.006 → 0.471, 500 images) and muharaf 0.046 → 0.600 (insertions 0.012 → 0.423, plus memorized reading lost). Avoid on presentation-slide and muharaf-style handwritten inputs. onlinekhatt and khatt (amad-vlm5's other training-overlap sets) also regress.
• Not yet shown to beat the cheaper fix. The natural control — amad-vlm5 with a repetition penalty and a think budget at inference — is being run; until it lands, merging is not proven necessary for this gain.
• One benchmark is a narrow window. These numbers cover transcription accuracy on 13 Arabic document styles. They say nothing about photographed pages in poor light, skew, mixed Arabic–Latin text, tables and layout, latency, or instruction following — where this model may hold unmeasured strengths and unmeasured regressions.

Reproduction

merge_method: ties
base_model: Qwen/Qwen2.5-VL-7B-Instruct
models:
  - model: amad-iq/amad-vlm5
    parameters: {weight: 1.0, density: 0.5}
  - model: AhmedZaky1/DIMI-Arabic-OCR-V2   # V2 only — V1 is subsumed by V2
    parameters: {weight: 1.0, density: 0.5}
parameters: {lambda: 1.0, normalize: false}
dtype: bfloat16

License

Apache 2.0, inherited from the base model and both parents.

Author: Murtadha · amad-iq