CMSManhattan/JiRackTernaryPrecision-2B-chat

JiRack Ternary Precision 8B (GGUF)

JiRack Ternary Precision 8B is a ternary (1.58-bit) language model by CMSManhattan (Konstantin Grabko) with 8.08B parameters, built for tool calling, agents and robotics software. Every linear layer of the transformer uses ternary weights {-1, 0, +1} with a per-tensor scale and 8-bit activations (BitNet b1.58 style). This repository has the model as GGUF for llama.cpp: 2.6 GB instead of 16.2 GB in bf16, runs on a CPU.

Quick start

HF Spaces demo chat https://huggingface.co/spaces/CMSManhattan/JiRack-2B-Chat

Files

File	Size	Notes
JiRackTernaryPrecision-8B-TQ2_0.gguf	2.6 GB	recommended; tested against the reference model (below)
JiRackTernaryPrecision-8B-TQ1_0.gguf	2.2 GB	same ternary weights, denser packing; not separately evaluated
jirack_gguf_proxy.py + jirack_openai_server.py		OpenAI-compatible endpoint with tool calls on top of llama-server
chat_template.jinja		the exact fine-tuning prompt format (also embedded in the GGUF)
eval/		tool-calling test logs (GGUF and reference model), BFCL summary

Model

Architecture	Qwen2-style decoder (DeepSeek-R1-Distill layout), 28 layers, hidden 3584, FFN 20480, 28 heads / 4 KV heads (GQA), bias on q/k/v, separate output head
Parameters	8.08B (196 ternary linear layers)
Initialisation	DeepSeek-R1-Distill-Qwen-7B, FFN widened 18944 -> 20480
Pretraining	phase 0: continued pretraining in bf16 on C4; phase 1: quantization-aware ternarization on C4 (lambda 0 -> 1); phase 2: ternary, C4 + Wikipedia
Fine-tuning (ternary, lambda 1)	main SFT (chat, reasoning, tool calling) -> robotics SFT (robotics code in Python / C++ / Java, BFCL-style tool data, and examples where none of 8-14 similar tools fits)
Tokenizer	DeepSeek-R1 / Qwen2 tokenizer with `<
Context	fine-tuned with sequences up to 32,768 tokens

GGUF export: every ternary weight is stored exactly as the model computes it (clamp(round(W / mean|W|), -1, 1) * mean|W|), then packed as TQ2_0 / TQ1_0.

Results

Internal tool-calling test (117 cases, greedy)

Category	Reference (PyTorch)	GGUF TQ2_0
Single call (one tool, all arguments given)	37/38	37/38
Parallel calls	17/19	17/19
Follow-up (answer from a tool result)	19/20	19/20
No tool needed	25/25	25/25
Clarify a missing argument	13/15	14/15
Total	111/117 (94.9%)	112/117 (95.7%)

Same 117 cases and the same prompt format in both runs; the GGUF file is served by llama-server through jirack_gguf_proxy.py. The GGUF file passes one more case than the reference (clarify 14/15 vs 13/15): a difference of one case in either direction is expected between the two backends, which do the arithmetic differently; the packed file keeps the model's quality.

BFCL v3 (reference model, greedy)

Run with evalscope 1.14.0 / bfcl-eval 2026.3.23 in function-calling mode. Totals are weighted by the number of cases, as evalscope reports them, so they are not comparable with the official leaderboard averages; multi-turn categories were not run. Raw tables are in eval/bfcl_summary.txt.

Non-live	N	Accuracy
simple	400	82.8%
multiple	200	82.5%
parallel	200	82.0%
parallel_multiple	200	69.0%
java	100	48.0%
javascript	50	52.0%
irrelevance	240	81.7%
Total	1,390	76.8%

Live (real user requests)	N	Accuracy
live_simple	258	50.4%
live_multiple	1,053	49.8%
live_parallel	16	43.8%
live_parallel_multiple	24	33.3%
live_relevance	18	72.2%
live_irrelevance	882	66.2%
Total	2,251	56.2%

For comparison, the ternary 16B scores 77.4% / 57.8% on the same sets: the 8B is within 1.6 points at half the size, and declines irrelevant requests more often (live_irrelevance 66.2% vs 61.3%).

Quick start (llama.cpp)

Plain chat:

llama-cli -m JiRackTernaryPrecision-8B-TQ2_0.gguf -c 8192 -cnv

Tool calling through an OpenAI-compatible API. llama-server returns the model's tool calls as text (<|tool_call_start|> {"name": ..., "arguments": ...} <|tool_call_end|>); the proxy builds the exact fine-tuning prompt and turns the calls into standard OpenAI tool_calls, so any OpenAI client or agent framework can use the model:

llama-server -m JiRackTernaryPrecision-8B-TQ2_0.gguf --port 8080 -c 32768 --special
python3 jirack_gguf_proxy.py --llama http://127.0.0.1:8080 --port 8001   # needs jirack_openai_server.py next to it

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8001/v1", api_key="none")
tools = [{"type": "function", "function": {
    "name": "get_weather", "description": "Current weather for a city",
    "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}]
r = client.chat.completions.create(model="jirack-8b", tools=tools,
                                   messages=[{"role": "user", "content": "Weather in Austin?"}])
print(r.choices[0].message.tool_calls)

Return tool results as tool messages; several calls in one turn (parallel calls) are supported. Use greedy decoding (temperature 0) for tool calls.

Speed (llama-bench, CPU only, 2x Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz, 48 threads): 16.4 tokens/s prompt processing (512 tokens), 9.0 tokens/s generation. This CPU lacks AVX2, which the TQ2_0 kernels use for speed.

Recommended use

1. Validate calls before executing them (tool exists, required arguments present).
2. Offer only the tools that fit the task; fewer, clearly named tools give the best accuracy.
3. Use a calculator / search tool for arithmetic and facts instead of the model's memory.

Limitations

• Weakest areas: real user requests with many similar functions (live_multiple 49.8%), several different calls in one turn (live_parallel_multiple 33.3%), and Java / JavaScript call syntax (48.0% / 52.0%).
• Sometimes calls a tool when none fits (live_irrelevance 66.2%); check calls before executing them.
• Ternary model: world knowledge is below that of the bf16 base model; not for medical, legal or financial advice.

Training data attribution

Pretraining uses C4 and Wikipedia. Fine-tuning data was generated for this project.

License

© 2025-2026 Konstantin Vladimirovich Grabko, all rights reserved.