CMSManhattan/JiRackTernaryPrecision-16B-chat

JiRack Ternary Precision 16B (GGUF)

JiRack Ternary Precision 16B is a ternary (1.58-bit) language model by CMSManhattan (Konstantin Grabko) with 15.9B parameters, built for tool calling, agents and robotics software. Every linear layer of the transformer uses ternary weights {-1, 0, +1} with a per-tensor scale and 8-bit activations (BitNet b1.58 style). This repository has the model as GGUF for llama.cpp: 4.8 GB instead of 31.8 GB in bf16, runs on a CPU.

Quick start

HF Spaces demo chat https://huggingface.co/spaces/CMSManhattan/JiRack-16B-Chat

Files

File	Size	Notes
JiRackTernaryPrecision-16B-TQ2_0.gguf	4.8 GB	recommended; verified against the reference model (below)
JiRackTernaryPrecision-16B-TQ1_0.gguf	4.1 GB	same ternary weights, denser packing; not separately evaluated
jirack_gguf_proxy.py + jirack_openai_server.py		OpenAI-compatible endpoint with tool calls on top of llama-server
chat_template.jinja		the exact fine-tuning prompt format (also embedded in the GGUF)
eval/		tool-calling test log of the TQ2_0 file

Model

Architecture	Qwen2-style decoder (DeepSeek-R1-Distill layout), 48 layers, hidden 5120, FFN 15360, 40 heads / 8 KV heads (GQA), bias on q/k/v, separate output head
Parameters	15.9B (336 ternary linear layers)
Initialisation	DeepSeek-R1-Distill-Qwen-14B, FFN widened 13824 -> 15360
Pretraining	phase 0: continued pretraining in bf16 on C4; phase 1: quantization-aware ternarization on C4 (lambda 0 -> 1 over 8,000 steps); phase 2: ternary, C4 + Wikipedia
Fine-tuning (ternary, lambda 1)	main SFT (chat, reasoning, tool calling) -> agentic SFT (SWE-rebench and Docker-sandbox coding trajectories, no-tool / clarify data) -> robotics SFT (robotics code in Python / C++ / Java, BFCL-style tool data, and examples where none of 8-14 similar tools fits)
Tokenizer	DeepSeek-R1 / Qwen2 tokenizer with `<
Context	fine-tuned with sequences up to 32,768 tokens

GGUF export: every ternary weight is stored exactly as the model computes it (clamp(round(W / mean|W|), -1, 1) * mean|W|), then packed as TQ2_0 / TQ1_0.

Results

Internal tool-calling test (117 cases, greedy)

Category	Reference (PyTorch)	GGUF TQ2_0
Single call (one tool, all arguments given)	36/38	36/38
Parallel calls	18/19	18/19
Follow-up (answer from a tool result)	20/20	20/20
No tool needed	25/25	25/25
Clarify a missing argument	13/15	12/15
Total	112/117 (95.7%)	111/117 (94.9%)

Both runs use identical prompts (checked on all 117 cases); the GGUF file loses one case.

BFCL v3 (reference model, greedy)

Run with evalscope 1.14.0 / bfcl-eval 2026.3.23 in function-calling mode. Totals are weighted by the number of cases, as evalscope reports them, so they are not comparable with the official leaderboard averages; multi-turn categories were not run.

Non-live	N	Accuracy
simple	400	82.5%
multiple	200	82.0%
parallel	200	82.5%
parallel_multiple	200	66.5%
java	100	59.0%
javascript	50	62.0%
irrelevance	240	80.8%
Total	1,390	77.4%

Live	N	Accuracy
live_simple	258	63.2%
live_multiple	1,053	53.4%
live_parallel	16	56.3%
live_parallel_multiple	24	54.2%
live_relevance	18	72.2%
live_irrelevance	882	61.3%
Total	2,251	57.8%

Quick start (llama.cpp)

Plain chat:

llama-cli -m JiRackTernaryPrecision-16B-TQ2_0.gguf -c 8192 -cnv

Tool calling through an OpenAI-compatible API. llama-server returns the model's tool calls as text (<|tool_call_start|> {"name": ..., "arguments": ...} <|tool_call_end|>); the proxy builds the exact fine-tuning prompt and turns the calls into standard OpenAI tool_calls, so any OpenAI client or agent framework can use the model:

llama-server -m JiRackTernaryPrecision-16B-TQ2_0.gguf --port 8080 -c 32768 --special
python3 jirack_gguf_proxy.py --llama http://127.0.0.1:8080 --port 8001   # needs jirack_openai_server.py next to it

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8001/v1", api_key="none")
tools = [{"type": "function", "function": {
    "name": "get_weather", "description": "Current weather for a city",
    "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}]
r = client.chat.completions.create(model="jirack-16b", tools=tools,
                                   messages=[{"role": "user", "content": "Weather in Austin?"}])
print(r.choices[0].message.tool_calls)

Return tool results as tool messages; several calls in one turn (parallel calls) are supported. Use greedy decoding (temperature 0) for tool calls.

Speed: TQ2_0 needs CPU kernels with AVX2 to be fast. On a 2013 dual Xeon E5-2697 v2 (no AVX2) we measured 7.9 tokens/s prompt processing and 3.8 tokens/s generation; modern CPUs and GPUs are much faster.

Recommended use

1. Validate calls before executing them (tool exists, required arguments present).
2. Offer only the tools that fit the task; fewer, clearly named tools give the best accuracy.
3. Use a calculator / search tool for arithmetic and facts instead of the model's memory.

Limitations

• Weakest areas: choosing one function from a long list of similar ones (live_multiple 53.4%), Java / JavaScript call syntax, and requests that need several different tools at once (parallel_multiple 66.5%).
• Sometimes calls a tool when none fits (live_irrelevance 61.3%).
• Ternary model: world knowledge is below that of the bf16 base model; not for medical, legal or financial advice.

Training data attribution

Agentic fine-tuning uses nebius/SWE-rebench-openhands-trajectories (CC BY 4.0). Pretraining uses C4 and Wikipedia. Other fine-tuning data was generated for this project.

License

© 2025-2026 Konstantin Vladimirovich Grabko, all rights reserved.