BEE-spoke-data/falcon-ocr-v1.5-cpu

Falcon-OCR v1.5: kernel-ready CPU files

Pre-packed weights of tiiuae/Falcon-OCR v1.5 (revision fe757d59) for the Rust CPU runner at github.com/amazingvince/falcon-ocr-cpu, plus a trained draft head that speeds up decoding.

Each model file holds every tensor in the layout the runner's kernels read and is memory-mapped in place. For the recorded INT16/INT8 comparison, loading takes about 10 ms against about 2 s to quantize the original FP32 checkpoint at start-up. These are not transformers checkpoints; use them with the runner above.

Files

File	What	Size
falcon-ocr-v1.5-fp32.safetensors	exact mode: FP32 body weights and FP32 KV cache	1,133 MB
falcon-ocr-v1.5-int16-g64.safetensors	near-exact mode (the default): 16-bit body weights with one scale per 64 weights; 16-bit KV cache	806 MB
falcon-ocr-v1.5-int8-gptq-g64.safetensors	fast mode: 8-bit GPTQ body weights (act-order, groups of 64); 8-bit KV cache	638 MB
falcon-ocr-v1.5-draft-head.safetensors	the draft head for speculative decoding (below); used automatically when it is in the model folder	94 MB
tokenizer.json, tokenizer_config.json, config.json	from the original model

Every model file also carries the FP32 embedding, vocabulary head, norms, projector and attention sinks, and an INT8 screen of the vocabulary head with its error bounds, which the runner uses to pick exactly the same greedy token as the full FP32 head. Their safetensors metadata records the source checkpoint hash, the recipe and a digest of every tensor (--verify-model-file checks it).

The filenames describe body weight storage: fp32, int16-g64, and int8-gptq-g64. INT16 uses scaled integer codes, not FP16/BF16. g64 means one FP32 scale per 64 body inputs; GPTQ INT8 is this runner's quantizer, not a GGUF encoding. Other model weights remain FP32, with an INT8 vocabulary-screen auxiliary.

The FP32 file uses falcon-ocr-kernel-fp32-v1; the INT16/INT8 files keep falcon-ocr-kernel-v1. Their metadata records the recipe and source checkpoint. The draft head has its separate format and role.

Upgrade the runner and download these files together. INT16 and INT8 replace the old near-exact and fast filenames, while --mode exact, --mode near-exact, and --mode fast keep their meanings. Older binaries need explicit --model-file paths for the renamed quantized files and cannot load the new FP32 format.

Usage

git clone https://github.com/amazingvince/falcon-ocr-cpu && cd falcon-ocr-cpu
cargo build --release --locked
python -m pip install huggingface_hub
hf download BEE-spoke-data/falcon-ocr-v1.5-cpu --local-dir models/falcon-ocr-cpu
target/release/falcon-ocr --model models/falcon-ocr-cpu run page.png --text              # near-exact
target/release/falcon-ocr --model models/falcon-ocr-cpu --mode fast run page.png --text  # fast
target/release/falcon-ocr --model models/falcon-ocr-cpu --mode exact run page.png --text # FP32
target/release/falcon-ocr --model models/falcon-ocr-cpu doctor --text                    # what it will do

Build prerequisites: Rust 1.94.0, a C/C++ compiler, CMake, and NASM on x86; see the runner README for platform-specific setup. The download command requires the Hugging Face Hub package installed above.

CPU targets are x86-64 with AVX2 or AVX-512 and aarch64 (NEON), with a scalar fallback. x86-64 has model-backed measurements; aarch64 has compilation and kernel-test coverage, without a model-backed run.

Accuracy

Against the FP32 model, teacher-forced along its outputs on 55 calibration pages (24,262 tokens):

Mode	Mean KL per token	Tokens where the greedy choice differs
exact (FP32)	0	0
near-exact	9e-8	1
fast	2.7e-4	63

On 118 fresh English OmniDocBench pages (figures the ground truth leaves out masked on every system's input), near-exact produced exactly FP32's tokens on every page. Fast mode matched FP32 token for token on 93 pages, and its character error rate against the ground truth was 0.11 points lower than FP32's overall, with no repetition loops: no measurable loss on printed English. On handwriting and degraded scans in the runner's 200-page held-out set, fast mode's 8-bit weights tipped some uncertain pages into repetition loops (the runner's repetition stop cuts them short); use near-exact for such pages. Details: docs/MODES.md.

Speed

Recorded on Ryzen 9 7950X (16 cores, DDR5), Windows 11, default flags, without the draft head. These are mode measurements, not new benchmarks of the repacked files:

Mode	Journal page (6,544 image tokens)	Decode per token
exact (FP32)	38 s	29 ms
near-exact	21.5 s	15 ms
fast	12.6 s	8.5 ms

With the draft head (used automatically when it is in the model folder), 16 English held-out pages, total time with the n-gram drafter → with the head, tokens identical:

Mode	n-gram drafts only	with the draft head	Decode vs no speculation
exact	531 s	393 s (-26%)	1.67×
near-exact	277 s	208 s (-25%)	1.87×
fast	173 s	136 s (-21%)	1.50×

--max-dimension auto (opt-in) picks 768, 1024 or 1536 pixels per page from image statistics and saves about a quarter of the CPU time on its development set. It changes the output and failed its held-out quality gate. A later figure-masked comparison was diagnostic and did not change that verdict (details).

The draft head

An EAGLE-3-style drafter: one decoder block over the features of the target's layers 2, 11 and 19 (text only), a 16,383-token draft vocabulary, trained first on next-token prediction and then on seven-step chains (weights 0.5^j). It proposes up to four tokens per step while its confidence is at least 0.35, and the model verifies them in one pass, so it never changes the output, only the speed. The runner uses it automatically when this file sits next to the model files and speculation is on (--drafter ngram or --speculate 0 turn it off).

Training data: about 45,000 English document pages, rendered as images and transcribed by the official Falcon-OCR vLLM image (greedy), after removing duplicates and every page that shares an image or a 13-gram of text with OmniDocBench. No non-commercial or evaluation-only data set was used. Sources, with the pages of the final training stage:

Source	Pages	License or terms
allenai/olmOCR-mix-1025	25,841	ODC-BY-1.0 (intended for research and educational use under Ai2's Responsible Use Guidelines)
pixparse/pdfa-eng-wds	12,000	Common Crawl terms of use and Digital Corpora terms of use
pixparse/idl-wds	7,000	UCSF Industry Documents Library terms
docling-project/DocLayNet-v1.2	1,937	CDLA-Permissive-1.0
biglam/loc_beyond_words	500	CC0-1.0
NoTeS-Bank/ICDAR_2025_Handwritten_Notes_Understanding_Challenge	392	Apache-2.0
PDFPages/zenodo-presentations-open (CC-BY-4.0 records only)	230	CC-BY-4.0
HumynLabs handwritten notes (math, physics, chemistry, biology, computer science)	169	CC-BY-4.0

The pipeline, the recipe and every measurement are in research/draft-head.

Rebuilding the model files

The FP32 file preserves the pinned checkpoint's parameter bytes and adds the precomputed vocabulary screen. INT16 and GPTQ INT8 are fresh repacks of the previously published files, with identical tensors, metadata values, profiles and v1 formats. The draft head is unchanged. The encoding names and FP32 loading support do not introduce new accuracy or speed claims.

To rebuild from the pinned FP32 checkpoint instead (which reruns GPTQ for fast mode), install the Python helper dependencies and restore the locked calibration pages as described in Development, then run:

python scripts/fetch_reference.py --output artifacts/model
cargo build --release --locked --bins
mkdir -p artifacts/repacked
target/release/falcon-ocr --mode exact pack --output artifacts/repacked/falcon-ocr-v1.5-fp32.safetensors
target/release/falcon-ocr --mode near-exact pack --output artifacts/repacked/falcon-ocr-v1.5-int16-g64.safetensors
BIN=target/release/falcon-ocr-eval bash tools/make_gptq_overlay.sh
target/release/falcon-ocr --mode fast pack --output artifacts/repacked/falcon-ocr-v1.5-int8-gptq-g64.safetensors

Choose a new output directory for each build; on Windows use the .exe binary paths. Detailed recipes and artifact requirements are in docs/DEVELOPMENT.md.

License and attribution

Derived from tiiuae/Falcon-OCR by the Technology Innovation Institute, released under the Apache License 2.0. The model files are re-encodings and quantizations of those weights, and the draft head was trained on that model's features and outputs; all are redistributed under the same license. They are not an official TII release. Cite the original work: Falcon OCR tech report. The draft head's training data remains under its sources' own terms (table above).