Vignesh-5756/Unlimited-OCR-mlx-fp16

Unlimited-OCR (DeepSeek-OCR) — MLX, fp16

Apple MLX conversion of baidu/Unlimited-OCR, a 3.3B-parameter vision-language model for document OCR, layout parsing, and grounding (SAM ViT-B + CLIP ViT-L encoders → linear projector → DeepSeek-V2 MoE LLM). Runs natively on Apple Silicon — no PyTorch or CUDA at inference time.

Converted with convert.py from the upstream bfloat16 checkpoint: MoE experts stacked to [n_experts, out, in] for batched gather_mm, Conv2d weights transposed OIHW → OHWI, stored in float16.

Use

git clone https://github.com/vignesh-kumar-v/mlx-ocr-unlimited
cd mlx-ocr-unlimited && pip install -r requirements.txt

# Single image
python ocr.py --model Vignesh-5756/Unlimited-OCR-mlx-fp16 --image doc.png --prompt "<image>Free OCR."

# Multi-page PDF -> markdown
python ocr.py --model Vignesh-5756/Unlimited-OCR-mlx-fp16 --pdf paper.pdf --output out.md

Fidelity & performance

Compared against the reference PyTorch/CUDA implementation on a 15-page arXiv PDF: 0.99 output similarity in fp16 (this repo), 0.9998 when run with --dtype fp32. Decodes at ~175 tok/s (single image) / ~120 tok/s (multi-page) on an M5 Pro, ~6 GB resident in fp16.

Notes

• Load the tokenizer with PreTrainedTokenizerFast (the project's loader does this): AutoTokenizer misroutes this checkpoint to LlamaTokenizer and corrupts the byte-level BPE.
• The checkpoint omits norm_topk_prob, which must default to False.

Code, tests, and implementation notes: https://github.com/vignesh-kumar-v/mlx-ocr-unlimited