J-Fraudster/dots.mocr-W4A16-AutoRound

dots.mocr W4A16 AutoRound Quantized

This repository provides production-grade W4A16 quantized weights for dots-studio/dots.mocr using Intel AutoRound.

Available Variants

• J-Fraudster/dots.mocr-W4A16-AutoRound: Native AutoRound format.
• J-Fraudster/dots.mocr-W4A16-GPTQ: Exported GPTQ format optimized for direct serving in vLLM.

---

Quantization Details

The quantization recipe was tuned for high accuracy and long-context multimodal parsing while preventing degradation of visual features:

• Bits / Weight Type: 4-bit weights, 16-bit activations (W4A16)
• Group Size: 32 (Symmetric: True)
• Vision Tower Preservation: quant_nontext_module=False (vision encoder preserved in full precision to retain document grounding accuracy)
• Tuning Recipe: 1,200 iterations, 512 samples, sequence length 4,096 with torch.compile acceleration.

---

Deployment with vLLM

KV Cache Precision Warning: Always keep --kv-cache-dtype set to auto or bfloat16. Do not use low-precision KV cache formats (such as fp8), as doing so will cause the model to fail to generate any output or return empty responses. For high-throughput document parsing, serve the AutoRound variant directly with vLLM:

vllm serve  J-Fraudster/dots.mocr-W4A16-AutoRound \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --chat-template-content-format string \
  --dtype bfloat16 \
  --kv-cache-dtype auto \
  --max-model-len 32768 \
  --max-num-seqs 128 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --enable-chunked-prefill

Benchmark Highlights (Original Base Model)

• olmOCR-Bench: 83.9 Overall
• OmniDocBench (v1.5): 0.031 TextEdit / 0.029 Read OrderEdit
• Native Structured Output: Direct image-to-Markdown, HTML tables, LaTeX formulas, and SVG generation.

Acknowledgements

• Base model developed by dots-studio.
• Quantized using Intel AutoRound.