Pebble-10M-Chat-GGUF

GGUF conversions of basically-ai/Pebble-10M-Chat (Apache 2.0).

IMPORTANT: patched llama.cpp required

Pebble uses a custom hybrid Mamba2 + attention architecture. These GGUFs carry general.architecture = "pebble", which upstream llama.cpp refuses to load.

Everything needed to run them lives in the support repo:

rootendpoint/basicallyai_llama.cpp_support

• llama.cpp-pebble.patch - adds the pebble architecture to llama.cpp (applies cleanly against upstream commit 0eadefe)
• basicallyai_to_gguf.py - standalone converter (numpy + safetensors only)
• numpy_reference.py - independent reference implementation used to verify correctness

Apply the patch, rebuild llama.cpp, then:

llama-cli -m pebble-10m-chat-f16.gguf

Chat template

This model's SFT format is not documented upstream; the best-matching format (lowercase role labels, determined by A/B testing) is embedded in the GGUF:

user: <message>
assistant: <response>

llama-cli and llama-server pick it up automatically.

llama-server -m pebble-10m-chat-f16.gguf --host 127.0.0.1 --port 8080
# web UI at http://127.0.0.1:8080

Files

Quant	Size	Type
f16	20.7 MB	F16
q8_0	11.1 MB	mostly Q8_0
q4_k_m	7.5 MB	mostly Q4_K_M

Verification

Outputs were cross-checked token-by-token against an independent pure-numpy reference implementation over multiple prompts (CPU and CUDA backends, and quantized KV cache). Identical greedy sequences up to genuine argmax ties.

Model

• 10M parameters, hidden 384, 8 layers (6 Mamba2 + 2 attention), ctx 512, vocab 2048
• SFT on 250M tokens of smol-smoltalk
• A research-scale model: expect toy-level output quality.