DogukanUrker/BTL-4-GGUF

BTL-4 — Q4_K_M GGUF

Q4_K_M quantization of badtheorylabs/BTL-4, a 35B-A3B MoE fine-tuned from deepreinforce-ai/Ornith-1.0-35B.

Upstream ships only IQ2_XXS (2.30 bpw). This is the 4-bit build, quantized and measured on a 12 GB consumer GPU.

file	size	bpw
BTL-4-Q4_K_M.gguf	20.1 GiB (21.5 GB)	4.97

token_embd and output are held at q8_0; everything else follows the stock Q4_K_M mixture. 733 tensors, converted from the bf16 source.

---

Measured on my hardware

RTX 3060 12 GB · Ryzen 5 5600 · 16 GB DDR4 · Debian · llama.cpp CUDA

decode	52.1 tok/s
prefill	816-896 tok/s
load time	~46 s (--load-mode none)
context	170,000 with q8_0 KV

Decode was sampled 12 times across an 1,837-token generation: every sample fell between 52.0 and 52.4 tok/s. Throughput at 8K context was 52.4 tok/s, so the 170K window costs about 0.7%.

Only 10 of 40 layers hold a growing KV cache and those use 2 KV heads, which is why 170K fits at all — roughly 10 KB/token at q8_0.

---

Run it

llama-server -m BTL-4-Q4_K_M.gguf \
  -ngl 99 --n-cpu-moe 24 -c 170000 -fa on --jinja -np 1 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  -b 2048 -ub 1024 \
  --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --presence-penalty 0.0 --repeat-penalty 1.0 \
  --reasoning-format deepseek

This is the config I actually run, not a tuned demo. Two notes:

• --reasoning-format deepseek is not optional. Without it, thinking lands in content and accumulates every turn, and long agent runs degrade. Verified: with the flag, reasoning_content and content separate cleanly.
• --temp 0.6 is my daily setting. Upstream published temp 1.0 / top_p 0.95 and took all of its benchmark numbers with those. Use 1.0 if you are trying to reproduce them.

Lower --n-cpu-moe if you have more VRAM, raise it toward 26–28 if you OOM. On 16 GB of RAM, run --load-mode none and expect the load to take ~46 s; the experts are ~93% of the weights and you do not want them paging off a SATA disk.

Requires a llama.cpp build with qwen3_5_moe support.

---

What I verified

• Loads and generates correct code
• Tool calling works — finish_reason: tool_calls, correct function and arguments, through the model's XML template with --jinja
• Reasoning separation works with --reasoning-format deepseek
• It is extremely verbose. An LRU cache class took 1,837 tokens for ~400 tokens of code. Reversing a linked list took 3,744 characters of reasoning for 315 characters of answer. At max_tokens: 400 it never finished thinking and returned empty content. Budget 2–4K minimum, more for hard problems.

What is missing

No MTP. The upstream config declares mtp_num_hidden_layers: 1 and the converter writes block_count = 41, but the checkpoint contains no mtp.* or layers.40.* tensors — I checked the safetensors index directly, both counts are zero. A stock loader then fails on blk.40.attn_norm.weight. This build was converted with --no-mtp. Do not pass --spec-type draft-mtp.

No vision. The upstream checkpoint has 333 vision tensors and a vision_config, but convert_hf_to_gguf.py drops them. This is a text-only build with no mmproj.

On the upstream benchmark claims

I have not verified them, and this card does not repeat them as facts.

Of the three headline numbers, only BFCL v4 (AST) has a published paired baseline: 69.2 → 73.5 on the base model, same harness, same decoding. SWE-bench Verified (78.4%) and LiveCodeBench v6 (66.1%) are reported with — for the base model, so the fine-tune's contribution to either is unmeasured.

Separately, upstream's IQ2_XXS build reports 94.1% behavioural retention measured on 118 short-form factual, grounded-extraction and false-premise items. That gate contains no coding and no tool-calling, which is what the model is for.

I plan to run both builds through the same coding and tool-use harness. Until then, treat every number above the horizontal rule as mine and everything in this section as unverified.

Quantization

Converted from badtheorylabs/BTL-4 bf16 with convert_hf_to_gguf.py --no-mtp, then llama-quantize --token-embedding-type q8_0 --output-tensor-type q8_0. No importance matrix. Everything ran in ghcr.io/ggml-org/llama.cpp:full on the box described above.

License

Apache-2.0, inherited from badtheorylabs/BTL-4. The original base model, deepreinforce-ai/Ornith-1.0-35B, is MIT.