JustANormalTinkerer/hayai-ocr-v2.5-nova

Hayai OCR v2.5 Nova

Hayai OCR is an ultra-lightweight (~150M parameter) vision-to-text model engineered for ultra-fast, crop-level transcription across Japanese, Chinese, Korean, and English.

Hayai couples Google's SigLIP2 NaFlex vision encoder with a custom 12-layer causal transformer decoder. It transcribes dense, stylized, horizontal, and vertical text in a single forward pass without requiring an intermediate text-line detection stage (e.g., DBNet/YOLO).

Note: Hayai is designed specifically for crop-level recognition. For full-page scanning, pair it with a text detector.

---

What’s New in v2.5

Hayai v2.5 retains the core backbone of v2.1 (SigLIP2 NaFlex ~86M + 12-layer GQA decoder) while introducing major structural and efficiency upgrades:

• 4x Token Reduction (DSC Projector): Features a new Downsampling Spatial Convolution (DSCProjector) that reshapes patch embeddings into a 2D grid and applies a 2x pixel unshuffle (4 patches -> 1 token).
• Lower Prefill Latency & KV Footprint: By reducing decoder vision tokens by 75%, v2.5 dramatically cuts prefill latency and memory footprint, making higher patch budgets computationally affordable.
• Learnable Residual Scaling: Decoder layers now incorporate learnable per-channel scaling parameters (attn_res_scale, ffn_res_scale, initialized at 1.0) to stabilize deep residual propagation.
• Engineered for High-Throughput Inference: generate() features static KV-cache allocation, precomputed 1D text RoPE, and FP16 autocasting on CUDA devices.
• Auxiliary IDS Co-training: Pre-trained with an auxiliary 226-class Ideographic Description Sequence (IDS) classification head to sharpen character-level discrimination across rare CJK glyphs (training only; zero overhead at inference).

Architectural Comparison

Component	v2.1	v2.5 Nova
Vision Projector	2-layer MLP (1 patch -> 1 token)	DSCProjector: Reshape -> Replicate Pad -> Pixel Unshuffle (4 -> 1) -> LayerNorm -> MLP -> RMSNorm
Decoder Vision Tokens	N patches	N / 4 tokens (2D mRoPE runs on compressed spatial grid)
Residual Connections	Standard additive: x + f(x)	Learnable per-channel scale: x + scale * f(x)
Auxiliary Supervision	None	226-class IDS Head (training-only for CJK grounding)
Inference Path	Standard dynamic greedy loop	Preallocated static KV-cache + precomputed RoPE caches
Decoder Architecture	12 layers, d_model=512, d_ffn=2048, 8 query / 2 KV heads, SwiGLU, QK RMSNorm	Identical core parameters

---

Quickstart

import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor, PreTrainedTokenizerFast

MODEL_ID = "JustANormalTinkerer/hayai-ocr-v2.5-nova" 

model = AutoModel.from_pretrained(MODEL_ID, trust_remote_code=True).cuda().eval()
tokenizer = PreTrainedTokenizerFast.from_pretrained(MODEL_ID)
processor = AutoProcessor.from_pretrained("google/siglip2-base-patch16-naflex")

image = Image.open("example.png").convert("RGB")

# Select patch budget: 256 (throughput), 384 (balanced), or 512 (quality)
inputs = processor(images=[image], max_num_patches=384, return_tensors="pt").to("cuda")

with torch.no_grad():
    texts = model.generate(
        pixel_values=inputs["pixel_values"],
        pixel_attention_mask=inputs["pixel_attention_mask"],
        spatial_shapes=inputs["spatial_shapes"],
        tokenizer=tokenizer,
        max_new_tokens=128,
        repetition_penalty=1.0,  # Keep at 1.0 (disabled) for OCR
    )

print(texts[0])

Requirements: trust_remote_code=True is required for custom block-causal attention and 2D multi-dimensional RoPE (mRoPE). Standard greedy decoding is recommended. For helper utilities, see hayai-ocr on GitHub.

---

Resolution Strategy (max_num_patches)

Because SigLIP2 NaFlex scales inputs dynamically preserving aspect ratio, max_num_patches governs your spatial budget. Thanks to the 4x patch compression in v2.5, higher patch counts run significantly faster than in earlier architectures:

max_num_patches	Vision Tokens (Decoder)	CER ↓	Exact Match ↑	Text-only CER ↓	Text-only EM ↑	Relative Latency*
256	<= 64	4.95%	75.35%	3.54%	82.65%	Baseline (1.00x)
384 (Default)	<= 96	3.65%	79.15%	2.36%	86.31%	~1.19x
512 (Max Quality)	<= 128	3.10%	80.68%	1.78%	88.04%	~1.34x

*Evaluated on JMangaBench_Mixed (3,286 crops) under NFKC normalization. Latencies measured on an NVIDIA T4 GPU.

Which patch budget should you use?

• 256 — Throughput-Oriented: Best for high-volume pipelines, wide-aspect crops, and clean horizontal text.
• 384 — Recommended Default: Excellent balance of speed and recognition fidelity. Closes over 70% of the accuracy gap to 512.
• 512 — Fine-Grained / Small Glyphs: Critical for dense panels, small stylized fonts, and complex vertical layouts (e.g., dense slices drop CER from 7.43% down to 2.70%).

---

Benchmarks

Crop-Level Recognition (JMangaBench_Mixed)

Evaluated across 3,286 standard benchmark crops under Unicode NFKC normalization:

Model	Parameters	CER ↓	Exact Match ↑	Text-only CER ↓	Text-only EM ↑
MangaOCR	~150M	4.68%	73.52%	2.70%	82.87%
BaberuOCR	~150M	4.59%	72.25%	2.60%	81.65%
PaddleOCR-VL-For-Manga	~900M	2.91%	78.91%	1.87%	84.66%
HayaiOCR v2.1	~150M	3.23%	79.67%	1.90%	87.46%
HayaiOCR v2.5 Nova (384)	~150M	3.65%	79.15%	2.36%	86.31%
HayaiOCR v2.5 Nova (512)	~150M	3.10%	80.68%	1.78%	88.04%

Block-Level Evaluation

Evaluated on the 109 annotated dialogue blocks (1,101 reference characters) from Ceynou/comictxt:

Model	max_num_patches	CER ↓	Exact Match ↑	Total Edit Distance
HayaiOCR v2.1	256	8.63%	66.06% (72/109)	95
HayaiOCR v2.5	256	8.63%	66.06% (72/109)	95
HayaiOCR v2.5	384	8.45%	66.06% (72/109)	93
HayaiOCR v2.5	512	7.45%	69.72% (76/109)	82

---

Training Methodology

Hayai v2.5 was trained in a four-stage curriculum:

1. Projector Warmup (20k steps): Backbone and decoder frozen; only the DSCProjector is trained on a private ~1M-sample multimodal OCR corpus.
2. End-to-End Pretraining (10k steps): All weights unfrozen across the ~1M-sample dataset.
3. Multilingual Alignment (4 epochs): End-to-end training on 60k diverse CJK-English samples (hayai-finetuning-dataset-with-korean).
4. Targeted Fine-Tuning (4 epochs): Final optimization on the ~1.82k curated manga crop dataset (hayai-finetuning-dataset-final-final).

During all of the 4 stages, the decoder was also trained on Wikipedia, Japanese-Dialogue-Dataset and AozoraBunko

---

Text Normalization Pipeline

To accurately reproduce benchmark metrics and clean up raw generations, use this normalization pipeline:

import re
import unicodedata

def normalize_text(text: str) -> str:
    if not text:
        return ""
    # Canonical NFKC decomposition/composition
    text = unicodedata.normalize("NFKC", str(text))
    text = re.sub(r'[\r\n\t]+', ' ', text)
    
    # Strip whitespace situated between CJK characters
    cjk_char = r'[\u4e00-\u9fff\u3040-\u30ff\u3400-\u4dbf\uac00-\ud7af]'
    text = re.sub(f'({cjk_char})\\s+({cjk_char})', r'\1\2', text)
    
    return re.sub(r'\s+', ' ', text).strip()

---

Best Practices & Limitations

• Repetition Penalty: Leave repetition_penalty=1.0. CJK languages rely heavily on legitimate character repetition (e.g., onomatopoeia ドキドキ, grammatical reduplication, or sequential numbers like 2校...1校). Penalties greater than 1.0 will degrade transcription accuracy.
• Crop-Level Design: The model is optimized for reading localized bubbles, text lines, and sound effects. Passing an entire page will not work.

---

Citations

@inproceedings{baek2026mangav26,
  title     = {{Manga109-v2026: Revisiting Manga109 Annotations for Modern Manga Understanding}},
  author    = {Baek, Jeonghun and Miyai, Atsuyuki and Onohara, Shota and Ikuta, Hikaru and Aizawa, Kiyoharu},
  booktitle = {Culture x AI Workshop at ICML 2026},
  year      = {2026},
}

@article{multimedia_aizawa_2020,
  author    = {Kiyoharu Aizawa and Azuma Fujimoto and Atsushi Otsubo and Toru Ogawa and Yusuke Matsui and Koki Tsubota and Hikaru Ikuta},
  title     = {Building a Manga Dataset "Manga109" with Annotations for Multimedia Applications},
  journal   = {IEEE MultiMedia},
  volume    = {27},
  number    = {2},
  pages     = {8--18},
  year      = {2020}
}

@inproceedings{baek2022COO,
  title     = {COO: Comic Onomatopoeia Dataset for Recognizing Arbitrary or Truncated Texts},
  author    = {Baek, Jeonghun and Matsui, Yusuke and Aizawa, Kiyoharu},
  booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
  year      = {2022}
}