JBruceAI/VoronoiReasoner-41M-IT-GGUF

VoronoiReasoner-41M-IT (Native C++ & Multi-Quant GGUF)

VoronoiReasoner-41M-IT is an ultra-compact 41.46M parameter instruction-tuned reasoning model designed for ultra-high-speed edge inference, local servers, and real-time interactive agents.

By combining continuous hypersphere embeddings with hybrid double-gated short convolutions and grouped-query attention, VoronoiReasoner delivers ultra-low latency (0.75 ms/tok) and blazing throughput (1,200 – 1,328+ tokens/sec) on single-batch ($B=1$) interactive generation with a memory footprint of only ~36 MB VRAM.

---

Key Highlights

• ⚡ Blazing Native C++ Speed: Generates at 1,200 to 1,328+ tokens/second on single-batch ($B=1$) with sub-millisecond response latency (0.75 ms/token) on modern RTX GPUs.
• 💾 Microscopic Memory Footprint: Quantized GGUF models range from 34 MB to 42 MB, consuming under 50 MB VRAM with full GPU offloading.
• 🛠️ Multi-Quantization Support: Native GGUF builds available in Q4_K_M, Q8_0, IQ4_XS, and F16.
• 🎯 Commonsense Reasoning: Achieves 30.00% HellaSwag accuracy (+5.00% over random baseline) in only 41M parameters.
• 💬 Instruction Tuned: Fine-tuned on multi-turn conversational data (HuggingFaceH4/ultrachat_200k) with ChatML format.
• 🐍 Official Python Library (voronoi-reasoner): Pure tensor-only batch inference by default, optional token streaming, and direct C++ acceleration.

---

Available GGUF Quantizations

Model File	Format	File Size	BPW (Bits/Weight)	Generation Speed (RTX 5070)	Recommended Use
voronoi_reasoner_41m_q4_k_m.gguf	Q4_K_M	36.58 MB	7.32 BPW	1,328.2 tok/s (0.75 ms)	Recommended: Best balance of speed & quality
voronoi_reasoner_41m_q8_0.gguf	Q8_0	42.50 MB	8.51 BPW	1,202.0 tok/s (0.83 ms)	Maximum Quality: Near-lossless 8-bit precision
voronoi_reasoner_41m_iq4_xs.gguf	IQ4_XS	34.47 MB	6.90 BPW	1,300+ tok/s	Ultra-Compact: Smallest 4-bit memory footprint

---

1. Run with Native High-Speed C++ Engine (> 1,300 tok/s)

Run directly with llama-cli or the standalone voronoi_infer C++ runner:

# Single-turn generation at 1,328+ tokens/second:
llama-cli -m voronoi_reasoner_41m_q4_k_m.gguf -ngl 99 -p "Explain quantum superposition in simple terms." -n 128 --chat-template chatml --repeat-penalty 1.15 --temp 0.7 -st

# Interactive chat:
llama-cli -m voronoi_reasoner_41m_q4_k_m.gguf -ngl 99 --chat-template chatml

Or using the dedicated voronoi_infer binary:

voronoi_infer -m voronoi_reasoner_41m_q4_k_m.gguf -p "What is the speed of light?" -n 64 --no-stream

---

2. Run with Python (voronoi-reasoner Library)

Install the wheel:

pip install https://huggingface.co/JBruceAI/VoronoiReasoner-41M-IT-GGUF/resolve/main/voronoi_reasoner-0.1.0-py3-none-any.whl

Python API (Fast Non-Streaming by Default):

from voronoi_reasoner import VoronoiModel

# Loads model weights
model = VoronoiModel.from_pretrained("JBruceAI/VoronoiReasoner-41M-IT-GGUF")

# Fast non-streaming generation (zero per-token string decoding overhead):
response = model.generate("What is the nature of gravity?", max_new_tokens=100)
print(response)

# Optional streaming (opt-in side feature):
for piece in model.generate("Tell me a short poem about stars.", max_new_tokens=100, stream=True):
    print(piece, end="", flush=True)

Direct C++ Acceleration from Python (> 1,200 tok/s):

from voronoi_reasoner import VoronoiCppEngine

engine = VoronoiCppEngine("voronoi_reasoner_41m_q4_k_m.gguf")
response = engine.generate("Explain black holes in one paragraph.")
print(response)

CLI:

# Single prompt execution
voronoi run "What is the speed of light?"

# Interactive terminal chat
voronoi chat

---

Speed & Latency Benchmarks (RTX 5070 12GB)

All tests measured strictly on single-batch ($B=1$) interactive token generation:

Engine / Framework	Quant Type	Single-Batch Speed ($B=1$)	Per-Token Latency	Memory (VRAM)
Native C++ Engine (llama-cli)	Q4_K_M	1,328.2 tok/s	0.75 ms	~36 MB
Native C++ Engine (llama-cli)	Q8_0	1,202.0 tok/s	0.83 ms	~42 MB
Custom C++ Engine (voronoi_infer)	Q4_K_M	446.9 tok/s	2.24 ms	~36 MB
Python C++ Wrapper (VoronoiCppEngine)	Q4_K_M	> 1,200 tok/s	< 0.85 ms	~36 MB
PyTorch (Cached Non-Streaming)	FP16	27.6 tok/s	~36.2 ms	~60 MB

---

Benchmark: HellaSwag Commonsense Reasoning

Evaluated on 400 validation samples:

Model Head Configuration	Zero-Shot Accuracy	Margin Over Random (25.0%)
Random Chance Baseline	25.00%	0.00%
Standard Tied LM Head	28.50%	+3.50%
Spherical Voronoi Head	30.00%	+5.00%
Q4 Quantized Voronoi Head	27.50%	+2.50%

---

Architecture Specifications

• Parameters: 41,458,944 (41.46M)
• Layers: 12 Hybrid Blocks (7 Double-Gated Causal Convolutions + 5 Grouped-Query Attention layers)
• Model Dimension ($d$): 384
• Feed-Forward Dimension ($d_{ffn}$): 768
• Attention Heads: 6 Query Heads, 2 Key/Value Heads (Head Dim 64)
• Vocabulary Size: 65,536 tokens (ChatML format)
• Context Length: 1,024 tokens (extendable to 4,096 tokens)
• Pretrained Data Scale: 1.7 Billion Tokens (Ultra-FineWeb-L3 + ultrachat_200k)