Titlehhhh/captchaocr-ppocrv6

CaptchaOCR PP-OCRv6 — a text recogniser for Minecraft anti-bot captchas

The weights are published. Read this before you use them.

This repository holds the actual weights of a fine-tune of PaddlePaddle/PP-OCRv6_medium_rec_safetensors, trained by us on the CaptchaOCR dataset — our synthetic Minecraft anti-bot captchas. The published weights are the weight interpolation (WiSE-FT style "soup") with α = 0.4 between the fine-tuned checkpoint (epoch 7 of a 20-epoch run) and the pretrained base model.

Measured on 7 captcha types that were never trained on (whole plugins held out, 3 500 images, exact match on the normalised text):

	exact match (median over the 7 types)	CER	exact match (micro, 3 500 images)
the weights published here (α = 0.4)	0.638	0.1503	0.6123
the untouched base model (α = 0)	0.588	0.1657	0.5646
the fine-tune on its own (α = 1)	0.480	0.2233	0.5480

The published weights are better than the base model: +5.0 points of median exact match and +4.8 points of micro. Two honest caveats, both important:

1. The best α was chosen on the same held-out set the result is measured on, so the gain is optimistic — on new held-out plugin styles it will be smaller. The curve is smooth (the gain holds over the whole range α ∈ [0.25, 0.6]), so this is not a single lucky point.
2. The fine-tune on its own (α = 1) is worse than the base (0.480 vs 0.588 median): on 13 training types it specialises and loses the base's generality. The interpolation is what makes the fine-tune a net win.

Every number above is a measurement, and the file it was measured in is named next to it.

RU: веса опубликованы. Это дообучение PP-OCRv6 на нашем синтетическом датасете капч; опубликована интерполяция весов с базой, α = 0.4 (WiSE-FT). На 7 удержанных типах (3 500 картинок): 0.638 exact (медиана) / 0.6123 micro против 0.588 / 0.5646 у нетронутой базовой модели. Само дообучение без интерполяции — 0.480, то есть хуже базы. Оговорка: лучшее α выбрано на том же удержанном наборе, поэтому прирост оптимистичен.

What it reads

Frames from the CaptchaOCR dataset — the kind of pictures the model is trained on. The caption gives the type and the gold answer from labels.jsonl (what the player has to type). The links are absolute: the pictures live in the dataset repository, not in this one.

haq/mapcha/map — 128×128 map, 4 Latin characters · answer s8TO	LagBug/XProtect/code — 130×130, white background, noise · answer EzhYNY	LimboFilter/symbols — 128×128, 3 Cyrillic characters · answer ЪМИ

What this model is

A fine-tune of PP-OCRv6 medium (PaddleOCR's text recogniser) on the CaptchaOCR dataset: the captchas that open-source Minecraft anti-bot plugins draw, rendered by this project's own Java generators (46 captcha types, 23 upstream plugins).

• Architecture: PP-OCRv6 medium recogniser — PP-LCNetV4 backbone + CTC head, transformers class PPOCRV6SmallRecForTextRecognition (model_type: pp_ocrv6_small_rec), 19 134 326 parameters, hidden size 192.
• Dictionary: the base model's own 18 710 classes (index 0 = CTC blank, space at the end).
• Input: the captcha text band (the strip of pixels that carries the answer), not the whole frame: resized to height 48, width scaled proportionally and right-padded to 320, BGR channel order, normalised as (x / 255 - 0.5) / 0.5.
• Output: per-timestep class probabilities; decoded greedily (CTC: collapse repeats, drop the blank). Greedy decoding is the right decoder here — the answers are random strings, so a language model has nothing to exploit.

The band is the whole game. On the 29 dataset types that carry a band, the base model scores acc_micro = 0.5517 on the band and acc_micro = 0.0345 on the whole frame — about 16× worse on the frame. A solver must crop first.

Measured results

The weights published here

The fine-tune ran for 20 epochs (4 080 steps) on 13 captcha types; the best checkpoint by held-out median exact match was epoch 7 (step 1 632). The published weights are that checkpoint interpolated with the pretrained base model, θ = 0.4 · θ_finetuned + 0.6 · θ_base.

α	median exact (7 types)	median CER	micro exact (3 500)	micro CER
0.00 — the pretrained base	0.588	0.1657	0.5646	0.2446
0.25	0.620	0.1483	0.5991	0.2132
0.40 — published	0.638	0.1503	0.6123	0.2004
0.50	0.614	0.1630	0.6123	0.1964
0.60	0.592	0.1773	0.6100	0.1951
0.75	0.542	0.2040	0.5949	0.1995
1.00 — the fine-tune alone	0.480	0.2233	0.5480	0.2181

Per held-out type, at the published α = 0.40 (exact match / CER):

held-out type	base (α = 0)	published (α = 0.4)	fine-tune alone (α = 1)
out/JSH32/Minecaptcha/map	0.938 / 0.015	0.950 / 0.013	0.954 / 0.012
out/jonesdevelopment/sonar/complex	0.388 / 0.291	0.484 / 0.219	0.446 / 0.223
out/jonesdevelopment/sonar/legacy	0.342 / 0.335	0.458 / 0.255	0.380 / 0.299
out/jonesdevelopment/sonar/standard	0.588 / 0.166	0.638 / 0.150	0.480 / 0.238
out/laforetbrut/are-you-sure/code	0.000 / 0.808	0.018 / 0.681	0.024 / 0.600
out_ru/BotFilter/map	0.730 / 0.090	0.740 / 0.084	0.560 / 0.152
out_ru/CaptchaVerifyZ/map	0.966 / 0.007	0.998 / 0.000	0.992 / 0.002
median over the 7 types	0.588	0.638	0.480
micro over 3 500 images	0.5646	0.6123	0.5480

How the split was made: whole plugins were held out (5 plugins, 7 types, 3 500 images) — no image of a held-out type was ever trained on. Never a random image split: several captcha types have only a few distinct answers, so a random split lets the exact gold answer of a test image appear in the training half.

The training set: 13 of the 46 captcha types (6 499 images with a band), taken from the published dataset Titlehhhh/captchaocr-dataset (23 000 PNG, 46 types × 500, commit 6248c5b9b6ce7eb6e917efd7aa3952892beb7cea) — the very same render that is published, not a different one. The 7 held-out types (3 500 images) come from the same repository.

The base model on the whole dataset (archived benchmark)

These three numbers were measured on an earlier, smaller render of the dataset (100 images per type), before the dataset repository was re-uploaded with the 500-per-type render (23 000 PNG) that this model was trained on. They are the base model's numbers, not this repository's, and they were not re-measured on the new render.

measurement	value	source
exact match (micro) on the 29 band-carrying types, band crop	0.5517	reports/bench_models/ppocrv6-medium.json
exact match (micro) on the same 29 types, whole frame	0.0345	reports/bench_models/ppocrv6-medium.json
exact match (micro) on all 46 types, band crop	0.4043	reports/bench_models/ppocrv6-medium.json

Other captcha models measured on this dataset

All foreign models were measured by one protocol (bench/bench_models.py): the picture is cropped by the answer band (roi), the input is normalised the same way for everybody, the greedy decode is the same, and the metric is exact match on the normalised text. The set is the 29 band-carrying types, 20 images per type (580 items), on the earlier 100-images-per-type render of the dataset.

model	band exact match	item set	source file
PaddlePaddle/PP-OCRv6_medium_rec_onnx	0.5517	29 types / 580 items	reports/bench_models/ppocrv6-medium.json
PaddlePaddle/PP-OCRv5_mobile_rec_onnx	0.4621	29 types / 580 items	reports/bench_models/ppocrv5-mobile.json
PaddlePaddle/cyrillic_PP-OCRv5_mobile_rec_onnx	0.4621	29 types / 580 items	reports/bench_models/ppocrv5-cyrillic.json
easyocr	0.3281	8 types / 64 items (the old legacy64 subset)	reports/bench_models/easyocr.json
anuashok/ocr-captcha-v3	0.2966	29 types / 580 items	reports/bench_models/anuashok-v3.json
graycat660/doctr-crnn-vgg16-bn-russian-onnx	0.2776	29 types / 580 items	reports/bench_models/doctr-crnn-ru.json
Graf-J/captcha-crnn-base	0.0828	29 types / 580 items	reports/bench_models/grafj-crnn-base.json
Graf-J/captcha-crnn-finetuned	0.0672	29 types / 580 items	reports/bench_models/grafj-crnn.json
Graf-J/captcha-conv-transformer-finetuned	0.0638	29 types / 580 items	reports/bench_models/grafj-convtr.json
emelnov/ocr-captcha-v4-mailru	0.0431	29 types / 580 items	reports/bench_models/emelnov-mailru.json
jtrecenti/txtcaptcha-crnn	0.0034	29 types / 580 items	reports/bench_models/txtcaptcha-crnn.json
microsoft/trocr-base-printed	no score — the harness failed: ValueError: You need to specify either 'text' or 'text_target'.	—	reports/bench_models/trocr-base-printed.json

For the same base model, the whole frame instead of the band gives 0.0345 — 16× worse. The cropper, not the model, decides this task.

Our own models — a different protocol, not comparable with the table above

our model	median exact (held-out types)	micro exact	held-out set	source
this repository: PP-OCRv6 + interpolation α = 0.4	0.638	0.6123	7 types / 3 500 images	reports/TRAINING_PPOCR_V6_BIG.md
our CRNN+CTC, trained from scratch	0.372	0.381	4 types / 1 999 images	reports/TRAINING_CRNN_BIG.md
our CRNN+CTC on its own training plugins	1.000	1.000	7 types — pure memorisation	reports/TRAINING_CRNN_BIG.md

These two tables are not one protocol and must not be read as one ranking. The foreign models were measured by the benchmark harness on 29 band-carrying types (580 items, 20 images per type); our two models were measured on their own held-out plugin splits with their own training pipelines — 7 types / 3 500 images and 4 types / 1 999 images respectively. Different type sets, different image counts, different pipelines: the numbers are not directly comparable.

Every number in this card comes from a file of the CaptchaOCR project repository — the measured results are in reports/, they are not typed by hand and nothing is extrapolated.

Provenance and licence

• Base model: PaddlePaddle/PP-OCRv6_medium_rec_safetensors — Apache-2.0, by PaddlePaddle.
• This model: that base, fine-tuned on the CaptchaOCR dataset published here: Titlehhhh/captchaocr-dataset (46 captcha types · 23 000 labelled PNG · 23 upstream plugins).
• Licence of the weights: Apache-2.0, inherited from the base model. The fine-tuning code and this card are released under the same terms.
• Licence of the dataset: not Apache-2.0 — the dataset is split by the licence of the upstream plugin of each captcha type. The dataset card has the per-type table, and the file LICENSE in the dataset repository spells the same table out in full.

How to use it

The model is a plain transformers text recogniser. transformers 5.x has no image-to-text pipeline for a CTC recogniser any more (that task name now belongs to vision-language chat models), so load it directly:

import torch
from transformers import AutoModelForTextRecognition, AutoProcessor
from PIL import Image

REPO_ID = "Titlehhhh/captchaocr-ppocrv6"

model = AutoModelForTextRecognition.from_pretrained(REPO_ID).eval()
processor = AutoProcessor.from_pretrained(REPO_ID)   # preprocessor_config.json + the dictionary

image = Image.open("captcha.png").convert("RGB")
x, y, w, h = (25, 77, 79, 35)        # the answer band: take it from labels.jsonl
band = image.crop((x, y, x + w, y + h))

with torch.no_grad():
    out = model(**processor(images=band, return_tensors="pt"))

print(processor.post_process_text_recognition(out)[0]["text"])

The processor does the whole preprocessing: resize to height 48 (width scaled proportionally), right-pad to 320, RGB→BGR, normalise as (x / 255 - 0.5) / 0.5, and the greedy CTC decode that drops the blank and collapses repeats. The dictionary (18 710 characters) travels in preprocessor_config.json and is also readable as character_list.txt.

The normalisation (x / 255 − 0.5) / 0.5 is written explicitly into preprocessor_config.json (image_mean = image_std = [0.5, 0.5, 0.5]). The upstream file does not mention it, and the transformers image processor would then fall back to the ImageNet statistics — which is not what these weights were trained with.

The band (roi) is not optional: on scene-like captchas the answer is one line among several, and the whole frame makes the model read the wrong line.

Reproducing

The training code lives in the CaptchaOCR project repository:

# 1. the fine-tune (13 training types, 5 plugins / 7 types held out)
.venv-gpu\Scripts\python.exe train\finetune_ppocr.py --out train\run_ppocr\big --epochs 20 --batch-size 32 --lr 1e-5 --head-lr 1e-5 --workers 6 --eval-every 1 --save-every 100 --max-minutes 36
# 2. the weight interpolation (7 values of alpha, the same evaluation)
.venv-gpu\Scripts\python.exe train\run_ppocr\big\soup_eval.py
# 3. the export into a Hugging Face folder
.venv-gpu\Scripts\python.exe train\run_ppocr\export_hf.py --ckpt train\run_ppocr\big\soup_best.pt --out train\run_ppocr\big\export_soup

train/finetune_ppocr.py fine-tunes PaddlePaddle/PP-OCRv6_medium_rec_safetensors on the dataset, holds out whole plugins (never a random image split: several captcha types have only a few distinct answers, so a random split lets the exact gold answer of a test image appear in the training half), and reports exact match and CER per type plus the median over types. train/ppocr_data.py holds the dataset loader and the preprocessing shown above.

The measured report of the run quoted above is reports/TRAINING_PPOCR_V6_BIG.md (the analysis) and reports/TRAINING_PPOCR_V6_BIG.json (the numbers), in the project repository.

Repository metadata

file	what
model.safetensors	the weights: θ = 0.4·θ_finetuned (epoch 7, step 1 632) + 0.6·θ_pretrained
config.json	the architecture — copied from the base model, unchanged by the fine-tune
preprocessor_config.json	the input geometry, the normalisation and the 18 710-character dictionary
processor_config.json	which processor class AutoProcessor has to build
character_list.txt	the same dictionary, one character per line, index 0 = blank
finetune_meta.json	the provenance of this export (source checkpoint, epoch, step, α)

There is deliberately no pipeline_tag: transformers 5.x removed the image-to-text pipeline task (it belongs to vision-language chat models now), and a recogniser has to be called through AutoModelForTextRecognition, as in the snippet above. library_name: transformers is declared because the repository really is a transformers model.

Limitations

• Only the text band is read. This is a recogniser, not a solver: it does not reverse strings, does not turn Russian number words into digits and does not compute arithmetic.
• No Cyrillic in the dictionary. The base model's 18 710 classes contain no Cyrillic at all, so the Cyrillic captcha types of the dataset cannot be expressed in this model's alphabet.
• The data is synthetic. Every image is a render of this project's own generators — faithful to the upstream algorithms, but not a screenshot of a live server.
• Generalisation is the hard part. A recogniser can memorise a plugin's style instead of learning the characters; only the type-held-out numbers mean anything.
• The headline number is optimistic. α = 0.4 was picked on the same 7 held-out types the result is reported on, and the best checkpoint was picked the same way. Treat 0.638 as an upper estimate on new plugin styles, not as a guarantee.
• One hard type stays hard. out/laforetbrut/are-you-sure/code is at 1.8 % exact match (CER 0.68): the answer band of that type does not carry a clean text line.
• The weights are an interpolation, not a clean fine-tune. model.safetensors is 0.4·θ_finetuned + 0.6·θ_pretrained; the pure fine-tune is in the project repository (train/run_ppocr/big/best.pt) and is worse on unseen plugin styles.

Citation

This model is a fine-tune of PP-OCRv6 by PaddlePaddle (Apache-2.0). The dataset is the CaptchaOCR dataset, re-rendered from open-source Minecraft anti-bot plugins. See the dataset card for the provenance and the per-type licence of every captcha type.