Hawkeye-4B

Hawkeye-4B reads degraded text: blurry, low-resolution, far-away text of the kind a smart-glasses or phone camera captures in everyday use. It is Qwen/Qwen3-VL-4B-Instruct fine-tuned on BSTRB (Blurry Scene Text Recognition Benchmark), from the paper Hawkeye4B: Reading Illegible Text with Vision Language Models (Tang and Shin, University of Michigan).

Code · Project page · Dataset

Usage

The simplest way is the hawkeye package, which applies the exact protocol used in the paper:

pip install "hawkeye[infer] @ git+https://github.com/byron123t/hawkeye"

from hawkeye import Hawkeye
print(Hawkeye("byron123t/Hawkeye-4B").read("label.jpg"))

With transformers directly (>= 4.57), reproduce the same protocol: an OCR system prompt, the image upscaled ×3 with bicubic interpolation (the processor then caps it at 1,003,520 pixels), the instruction before the image, greedy decoding.

import torch
from PIL import Image
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

SYSTEM_PROMPT = ("You are an OCR engine. Transcribe the text visible in the image exactly as it appears.\nOutput rules:\n- Output ONLY the transcribed text. No preamble, no explanation, no commentary, no closing remarks.\n- Do not describe the image. Do not say what the text is or where it is.\n- Preserve the original line breaks: one visual line of text per output line.\n- Do not wrap the output in quotes or code fences, and do not add Markdown.\n- Do not correct spelling or expand abbreviations; reproduce what is printed.\n- If no text is legible, output nothing.")

model_id = "byron123t/Hawkeye-4B"
model = Qwen3VLForConditionalGeneration.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)

image = Image.open("label.jpg").convert("RGB")
image = image.resize((image.width * 3, image.height * 3), Image.BICUBIC)   # x3 bicubic

messages = [
    {"role": "system", "content": [{"type": "text", "text": SYSTEM_PROMPT}]},
    {"role": "user", "content": [{"type": "text", "text": "Transcribe all the text in this image."},
                                 {"type": "image", "image": image}]},
]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=4096, do_sample=False)
print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])

For exact reproduction of the reported numbers the evaluation also re-encodes each image as JPEG (quality 92, longest side at most 2,048 px) before the ×3 upscale; hawkeye does this. The shipped generation_config.json is greedy with up to 4,096 new tokens, as in the paper.

Results

Held-out BSTRB test split as in the paper (1,331 video frames from 224 videos, 241 photos, human-verified labels; the public split omits one video for privacy, which changes these scores by at most 0.001). Differences are paired against the base model on the same items, 95% bootstrap CIs clustered by video, Holm-corrected across all comparisons.

	Hawkeye-4B vs. base Qwen3-VL-4B
BSTRB video, BoW F1	0.627 → 0.728 (+10.1 pp, 95% CI [+8.7, +11.6])
BSTRB photos, BoW F1	+6.0 pp [+3.2, +8.6]
10 general OCR benchmarks, BoW F1	4 of 10 better, 6 no significant change, 0 worse (HierText +12.9, COCO-Text +5.8, MapText +4.7, RIMES +1.8 pp)
Smart-glasses captures (37 objects, out of distribution)	BoW F1 +14.7 pp, aligned F1 +16.6 pp (word inclusion +6.4 pp, not significant)
Snellen charts, text ≤ 7 px (character accuracy)	+7.2 pp [+4.5, +9.9]
Repetition loops, video text ≤ 20 px	14.3% → 7.2%

Against 20 API models (GPT, Kimi, Qwen3-VL-235B, Llama 4, Gemma, …) on BSTRB video, its BoW F1 is 0.728, second only to Hawkeye 8B among 24 models (Qwen3-VL-235B 0.726, Kimi K2.6 0.723, GPT-6 Astra 0.709). On BSTRB photos the largest API models remain ahead.

Training

1. SFT on the BSTRB training video frames (6,734 frames from 1,133 videos; the public split omits 30 of them for privacy), vision encoder unfrozen, learning rate 1e-6 (LLM and ViT) / 1e-5 (projector), inputs upscaled ×3.
2. GRPO with LoRA (rank 64, alpha 128, on q/k/v/o; peak learning rate 2e-5), reward = bag-of-words F1 against the training transcription, 300 steps; the LoRA is merged into the released weights.

Scripts and exact settings: github.com/byron123t/hawkeye.

Limitations

• Trained for English-dominant consumer packaging, signs, books and screens; non-Latin scripts are rare in the training data.
• Training labels are GPT-5 transcriptions (human-verified labels exist only for val/test).
• The gain in precision mostly comes from fewer repetition loops; on outputs that do not loop, the share of words not in the reference is similar to the base model's. Its most common failure is stopping early on dense text.
• On very small or unreadable text it can produce plausible but wrong completions (e.g. nutrition values); do not rely on it where errors matter (medication, legal text) without verification.

License

Apache-2.0, as is the base model. The BSTRB dataset used for training has its own license (CC BY-NC 4.0, gated).

Citation

@article{tang2026hawkeye,
  title   = {Hawkeye4B: Reading Illegible Text with Vision Language Models},
  author  = {Tang, Brian and Shin, Kang G.},
  journal = {arXiv preprint arXiv:XXXX.XXXXX},
  year    = {2026}
}