MoraxCheng/ohmyshot-ppformulanet-plus-l-coreml

PP-FormulaNet_plus-L and PP-DocLayoutV3 for Core ML (OhMyShot model pack)

PaddlePaddle's PP-FormulaNet_plus-L, which reads a picture of a formula as LaTeX, and PP-DocLayoutV3, which finds the formulas among text, converted to Core ML ML programs with 8-bit weights for OhMyShot, a screenshot app for macOS, which downloads them as a model pack: the text it recognizes then holds each formula as LaTeX, and Capture Formula copies the formula in a selection. The weights are PaddlePaddle's, quantized; see LICENSE.txt for the license and what was changed.

File	What it is
encoder.mlpackage	Input pixel_values, 1 × 3 × 768 × 768; outputs cross_k and cross_v, 8 × 1 × 16 × 144 × 32: each decoder layer's cross-attention keys and values.
decoder.mlpackage	One step: inputs ids (1 × 1, the last token), pos (1, its position), mask (1 × 1 × 1 × 1024, 0 for positions up to pos, -∞ after), cross_k, cross_v; output next, the likeliest next token, and logits. States k_cache and v_cache hold the self-attention keys and values.
layout.mlpackage	Input pixel_values, 1 × 3 × 800 × 800 (the picture scaled to it, value / 255); outputs logits, 1 × 300 × 25, and boxes, 1 × 300 × 4 (center x, center y, width, height, 0–1). Class 5 is a formula on a line of its own, 15 one within text.
tokens.txt	The tokenizer's tokens, one a line in id order (GPT-2 byte-level characters); ids below 23 are special.
config.json	Pre- and post-processing: inversion of dark pictures, cropping, scaling, normalization, the start and end tokens, the layout model's classes and threshold.
LICENSE.txt	Apache License 2.0, with what was changed.

Decoding starts from token 2 at position 0 and ends when the decoder gives 2. Run on the CPU (.cpuOnly).

Made with Tools/model-packs/formula_pack.py from these upstream revisions:

• https://huggingface.co/PaddlePaddle/PP-FormulaNet_plus-L_safetensors/tree/6cc004a8e91ea3f3819b05317f40b290cbe99c2b
• https://huggingface.co/PaddlePaddle/PP-DocLayoutV3_safetensors/tree/e4f489dc536556fc1be02b973ba06756e0d4a2ac

Checked against the OhMyShot formula corpus (Tools/l3-spike: 40 formulas drawn with LaTeX): read through these models on the CPU, with the app's own decoding, the share whose LaTeX draws exactly the formula drawn, by picture: 95.0% at 2x (192 dpi), 82.5% at 1x (96 dpi), 95.0% in dark mode (inverted first, as config.json has it), 75.0% in STIX Two Math, and 90.0% in TeX Gyre Pagella Math.

The layout model, on the CPU, against transformers' model and post-processing on 17 screen-sized pictures (Tools/model-packs/layout_pages.py: web pages, a chat in dark mode, Chinese, a paper; and screenshots without formulas): 28 of their 28 formulas found, 0 more, every box within IoU 0.96 of transformers'.