parseq-b-cyrillic-handwritten

A PARSeq-B (permuted autoregressive sequence) line recognizer for single-line handwritten Ukrainian/Cyrillic text — a larger ViT-B encoder variant (~96M parameters, ~4× the PARSeq-S model).

TL;DR

	value
Architecture	PARSeq — ViT-B encoder + 1-layer permutation-LM decoder (ECCV 2022)
Parameters	~96M
Stage	mixed pretrain (general handwriting reader, from scratch; pre-fine-tune)
Languages	Ukrainian
Decode	AR greedy + 1 refinement iteration
Best CER / WER	0.0118 / 0.0476 (greedy+refine, seed 42)
Input	a single cropped text line, BGR (numpy) — resized to 48×512
Output	the transcribed string (232-char frozen Cyrillic charset)

Intended use

Recognizing single cropped lines of handwritten Ukrainian text, downstream of a line/region detector or on already-segmented lines. Useful directly as a general handwriting reader, or as an init for fine-tuning on a target domain.

A companion ViT-S variant (~24M params) trained with the same recipe is available at Hukyl/parseq-s-cyrillic-handwritten.

How to use

Note: the checkpoint is a plain torch.save archive, not transformers-compatible. The payload is a dict:

import torch

payload = torch.load("best.pt", map_location="cpu", weights_only=True)
# keys: "model_state" (state_dict), "charset" (str), "config" (dict),
#       "metrics" (dict), "epoch" (int)
print(payload["config"])
# {'img_height': 48, 'img_width': 512, 'patch_size': (4, 8), 'embed_dim': 768,
#  'enc_num_heads': 12, 'enc_mlp_ratio': 4, 'enc_depth': 12, 'dec_num_heads': 12,
#  'dec_mlp_ratio': 4, 'dec_depth': 1, 'max_label_length': 100, 'dropout': 0.1,
#  'decode_ar': True, 'refine_iters': 1, 'drop_path_rate': 0.1}

To run it: construct a PARSeq model (ViT encoder + 1-layer two-stream permutation decoder; see baudm/parseq, Apache-2.0) with the embedded config, build a tokenizer over the embedded charset (token order is [E] + the charset string + [B] + [P]; [E] is id 0), then model.load_state_dict(payload["model_state"]).

The input geometry ships in the checkpoint's config: each crop is resized unconditionally to 48×512 RGB (no aspect preservation; INTER_AREA on downscale) and normalized (x − 0.5) / 0.5. Output confidence is the mean per-step max-softmax over the decoded sequence.

Architecture

PARSeq, after Bautista & Atienza, "Scene Text Recognition with Permuted Autoregressive Sequence Models" (ECCV 2022, arXiv:2207.06966). Model code is derived from baudm/parseq (Apache-2.0). Geometry, decoder, and charset are identical to the ViT-S variant; only the encoder width and head count are scaled up.

component	value
encoder	ViT-B — 12 layers, dim 768, 12 heads, MLP ratio 4, patch 4×8
decoder	1-layer two-stream permutation-LM, 12 heads, MLP ratio 4
training	K=6 permutations (permuted AR sequence modeling)
decode	autoregressive greedy + 1 refinement iteration
dropout / drop-path	0.1 / 0.1
input	48×512 RGB, unconditional resize, (x−0.5)/0.5
max label length	100
parameters	~96M

Charset

A frozen 232-character Cyrillic charset (sha256 d1b9161e…3ff976). Token IDs follow string order; specials are [E] (id 0) first and [B], [P] last. Text is not NFKD-normalized. The set covers the full Ukrainian Cyrillic block (incl. і ї є ґ and the apostrophe), the Russian-only capitals Э Ё and letters ъ ы ё, digits, punctuation, and a small set of Greek/math symbols. Latin lookalikes that are visually identical to Cyrillic letters (a c e i o y; A B C E H K M O P T X) are deliberately excluded — the target output space is Cyrillic-normalized, so those are folded to their Cyrillic counterparts.

Training

Trained from scratch on ~0.65M train crops materialized over a real + synthetic Cyrillic mix; synthetic:real ≈ 1.15:1. All sources are charset-filtered to the frozen vocab; every crop is treated as a handwritten line. The mix matches the ViT-S variant except for a smaller synthetic cap, to keep the heavier model's wall-time budget comparable.

Training was halted at epoch 24 of a planned 35; the published weights are the best-val-CER epoch reached (24), and the planned SWA weight-averaging phase (final 25% of epochs) did not run.

component	language	setting	≈ contribution
ukrhandwritten	uk	×3	~111k real lines
cyrillic_kaggle (Cyrillic Handwriting Dataset)	ru	×1	~74k real word/phrase crops
school_notebooks_RU	ru	cap 120k	120k real word crops
pumb-ai/synthetic-cyrillic-large	uk+ru	cap 250k	250k synthetic lines
nastyboget/synthetic_cyrillic_large	ru	cap 100k	100k synthetic lines

Hyperparameters (as launched)

hyperparameter	value
epochs	35 planned (halted at 24)
effective batch size	512 (micro-batch 128 × accum 4)
learning rate	5e-4 peak
schedule	OneCycleLR, warmup 10%
optimizer	AdamW, weight_decay 0.05
label smoothing	0.1
drop-path rate	0.1
permutations (K)	6
gradient clipping	20
precision	bf16 AMP
val fraction	0.02 (hash-stable holdout)
seed	42

Online data augmentation

Online augmentation was applied during training (default profile). Only training crops were augmented — the held-out 2% was never augmented — and every train crop was augmented (identity_prob = 0.0), at the 48×512 working resolution (downscaled before augmenting).

Each crop was transformed once per epoch by one geometric + one or two photometric operations at random.

pool	transforms (handwritten line crops)
geometric (pick 1)	margin pad 0.02–0.15; width-adaptive rotation (≤±8°); horizontal shear ±5°; elastic distortion (α30, σ4.5); baseline warp (amp 4–11% of height, wavelength 250–900 px)
photometric (pick 1–2)	paper-colour shift; Gaussian noise σ5–15; JPEG q25–65; contrast 0.7–1.2 / gamma 0.6–1.3; morphological erode/dilate (k≤4/3); bleed-through (α0.05–0.12)

Results

In-mix held-out validation (41,136 crops), greedy + 1 refinement iteration:

metric	value
CER	0.0118
WER	0.0476
exact-match accuracy	0.9270
n_samples	41,136

Limitations & biases

• Single-line handwriting crops only; a general handwriting reader, not tuned to any specific document domain — expect higher error on noisy document scans until fine-tuned.
• The charset is Cyrillic-normalized: Latin lookalikes are folded to their Cyrillic twins, so the model never emits the Latin forms.
• The training run was halted early (see Training), so the checkpoint may be short of the recipe's full potential.
• Single seed and validation split — no across-run variance estimate.

Training data & attribution

dataset	source	license	role
UkrHandwritten	Kaggle annyhnatiuk/ukrainian-handwritten-text	CC BY-SA 4.0	real Ukrainian lines (×3)
Cyrillic Handwriting Dataset	Kaggle constantinwerner/cyrillic-handwriting-dataset	CC0	real Russian word/phrase crops
school_notebooks_RU	HF ai-forever/school_notebooks_RU	MIT	real Russian word crops (cap 120k)
pumb-ai/synthetic-cyrillic-large	HF pumb-ai/synthetic-cyrillic-large	Apache-2.0	synthetic uk+ru lines (cap 250k)
nastyboget/synthetic_cyrillic_large	HF nastyboget/synthetic_cyrillic_large	MIT	synthetic Russian lines (cap 100k)

Attribution is mandatory for UkrHandwritten (CC BY-SA 4.0, share-alike) — please keep this credit if you redistribute or build on this model. Review each dataset's card for its own terms before redistribution.

License & lineage

Model weights are released under CC BY-SA 4.0, inherited from the UkrHandwritten training data (CC BY-SA 4.0, attribution + share-alike — the most restrictive of the training inputs). The PARSeq model code is Apache-2.0 (baudm/parseq).