RockMan256/granite-docling-258M-GGUF

Granite Docling 258M GGUF

GGUF conversion of RockMan256/granite-docling-258M, a fine-tuned version of ibm-granite/granite-docling-258M.

Files

File	Description
granite-docling-258M-bf16.gguf (317 MB)	LLM weights in BF16
mmproj-model-f16.gguf (182 MB)	Vision projector (mmproj) in F16

Usage with llama.cpp

# Build llama.cpp (if not already built)
cd llama.cpp && make -j

# Run with multimodal (image + text) inference
./llama-mtmd-cli \
  -m granite-docling-258M-bf16.gguf \
  --mmproj mmproj-model-f16.gguf \
  --chat-template chatml

Example prompts

# Describe an image
echo "Describe this image in detail." | ./llama-mtmd-cli \
  -m granite-docling-258M-bf16.gguf \
  --mmproj mmproj-model-f16.gguf \
  -p /path/to/image.png

# Text-only inference (no vision)
echo "What is the capital of France?" | ./llama-mtmd-cli \
  -m granite-docling-258M-bf16.gguf \
  -p - --chat-template chatml

Conversion Details

• LLM: Converted from user's fine-tuned RockMan256/granite-docling-258M (original weights, not remote)
• mmproj: Converted from ibm-granite/granite-docling-258M (vision encoder/connector unchanged from base)
• Quantization: BF16 (LLM), F16 (mmproj)

License

Apache 2.0 (same as base model)