Nicolassuez/Falcon-OCR-int8-openvino

Falcon-OCR, INT8 OpenVINO IR for CPU serving

The tiiuae/Falcon-OCR decoder rebuilt as an OpenVINO IR with INT8 weights, for serving document OCR on a CPU with no GPU anywhere. 478 MB of artifact against 1.08 GB of FP32 weights.

This is not a repackaging of the published checkpoint. The published code targets CUDA: it calls FlexAttention with a BlockMask, uses a Triton kernel for its gated feed forward and torch.compiles every layer. This IR recomputes the same function with operations a CPU runtime understands, and changes two things on purpose.

The attention sink became one more key. The checkpoint rescales every attention output by sigmoid(lse - sink), and lse only exists inside FlexAttention. That is exactly a softmax over the real keys PLUS one key whose logit is the sink and whose value is zero. So the queries carry a constant 1 in one extra lane and the keys are 65 wide instead of 64 (values stay 64 wide over 8 heads, since nothing rotates them). The model becomes an ordinary scaled dot product attention with no S x S matrix, and the sink is seeded at index 0 of the KV cache by the host, which removes any difference between prefill and decode.

The KV cache is an INPUT, not model state. Measured per attention layer at 3000 cached tokens on an AVX2 laptop: OpenVINO state plus SDPA 42.5 ms, explicit attention 38.7, preallocated state written in place 29.4, host-owned cache that the graph only reads 11.9, the two matrix products alone 10.0. A stateful Concat rebuilds the whole cache on every token. So both graphs take the cache as an input and return only the new token's keys and values, a few kilobytes.

Files

file	what it is
decoder_prefill.xml / .bin	the prompt pass, batch 1, fused SDPA
decoder_decode.xml / .bin	one token per call, DYNAMIC batch, explicit scored attention
embedder.xml / .bin	token and image-patch embedding
sink_state.npz	the per-layer, per-head sinks, pre-scaled, seeded at cache index 0
freqs_golden.npy	the spatial rotary frequencies
build.json	the quantization mode and the exact upstream revision
config.json, tokenizer.json, tokenizer_config.json	copied from the upstream checkpoint

Weights are compressed with NNCF INT8_ASYM, no calibration set. At run time the OpenVINO CPU plugin executes the linear layers on oneDNN with VNNI and dynamic activation quantization (inner_product,brgemm:avx2_vnni, wei:u8, src_dyn_quant_group_size:32). Attention stays floating point.

Quality against the FP32 reference

No content lost: 16 of 16 expected fragments on four pages. Three of the four pages are identical to the FP32 output at the bit level; the fourth differs in FORMAT (markdown against an HTML table) on a logit margin of 0.0997, which is the model's own near-tie. Keeping the embedding table and the output projection in FP32 does not repair that, and doubles the size.

The page must stay readable: on a dense A4 report with 12 px body text, 640 px reads 14 of 15 expected fragments and 512 px reads 5 of 15. Below roughly 640 px the body text falls under the patch grid and the model reads a page that is not there, at full speed and without any error. Note when scoring: the model writes 4.17 where the page has 4,17, so a literal comparison counts correct readings as failures.

How it is meant to be run

The decode graph takes a dynamic batch because that is where the speed is: a decode step reads all 220 MB of weights to advance ONE token, so rows share that read. Measured on the decode round at 350 cached positions, six threads: 1 row 30.8 ms, 6 rows 54.5, 12 rows 82.5, 24 rows 130.2. Cutting a page into balanced regions and decoding them together is what makes a dense page affordable, and merging several requests into the same round is what makes a queue of documents affordable.

Two traps the host must avoid, both of which produce fluent, degenerate text rather than an error: ov.Tensor(numpy_array) COPIES by default, so the cache must be wrapped with shared_memory=True; and a sequence length read from key_cache.shape[2] is frozen by tracing, so slice from the end.

A complete CPU service built on this IR, with the region layout, the batching scheduler, fused CPU kernels and an INT8 KV cache, lives in docker/models/falcon_ocr_cpu of the project this artifact was produced by.

Provenance and licence

Built from tiiuae/Falcon-OCR at revision 0c9f85d21adf928b3b34d37dba2f5be38048a1c4, exported and quantized with OpenVINO and NNCF. Apache 2.0, inherited from the upstream model. All measurements above were taken on an Intel i7-13700H with six inference threads, OpenVINO 2026.3.1, one stream, and are properties of that host as much as of this artifact.