Instructions to use CMSManhattan/Llama3Ternary_10b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use CMSManhattan/Llama3Ternary_10b with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf CMSManhattan/Llama3Ternary_10b:Q8_0
# Run inference directly in the terminal:
llama cli -hf CMSManhattan/Llama3Ternary_10b:Q8_0

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf CMSManhattan/Llama3Ternary_10b:Q8_0
# Run inference directly in the terminal:
llama cli -hf CMSManhattan/Llama3Ternary_10b:Q8_0

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf CMSManhattan/Llama3Ternary_10b:Q8_0
# Run inference directly in the terminal:
./llama-cli -hf CMSManhattan/Llama3Ternary_10b:Q8_0

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf CMSManhattan/Llama3Ternary_10b:Q8_0
# Run inference directly in the terminal:
./build/bin/llama-cli -hf CMSManhattan/Llama3Ternary_10b:Q8_0

Use Docker

docker model run hf.co/CMSManhattan/Llama3Ternary_10b:Q8_0

LM Studio
Jan

vLLM

How to use CMSManhattan/Llama3Ternary_10b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "CMSManhattan/Llama3Ternary_10b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CMSManhattan/Llama3Ternary_10b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/CMSManhattan/Llama3Ternary_10b:Q8_0

Ollama
How to use CMSManhattan/Llama3Ternary_10b with Ollama:
```
ollama run hf.co/CMSManhattan/Llama3Ternary_10b:Q8_0
```
Unsloth Desktop
Docker Model Runner
How to use CMSManhattan/Llama3Ternary_10b with Docker Model Runner:
```
docker model run hf.co/CMSManhattan/Llama3Ternary_10b:Q8_0
```

Lemonade

How to use CMSManhattan/Llama3Ternary_10b with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull CMSManhattan/Llama3Ternary_10b:Q8_0

Run and chat with the model

lemonade run user.Llama3Ternary_10b-Q8_0

List all available models

lemonade list

Atomic Chat

Llama 3 Ternary 10B

Built with Llama

Ternary (1.58-bit) derivative of Llama 3 8B with a widened FFN, packed for BitNet.cpp. It does not load in stock llama.cpp.

Inventor: Konstantin Vladimirovich Grabko
Organization: CMS Manhattan
Site: cmsmanhattan.com

Llama 3 is licensed under the Llama 3 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved.
A copy of that license ships in this repository as LICENSE.

What this is

Llama 3 8B weights with an expanded FFN, so the parameter count is about 10B, then quantized to ternary {-1, 0, 1}. Inference is meant for Microsoft BitNet.cpp kernels (AVX2 / AVX-512), not for stock llama.cpp.

Use this checkpoint as a small, low-RAM chat model. Domain adaptation (your data, your facts) is a separate paid job — write to grabko@cmsmanhattan.com.

Fine-tune with LoRa

Cheap fine-tune with LoRa for home or small companies
Easy LoRa is best way to adapt for you domain

Service option

Welcome to retrain on your domain with QAT/SFT and pack to Microsoft Bitnet kernels on llama fork on bitnet.ccp
So BitNet.cpp much faster than llama.cpp
More options with pre-train to raise more quality and more level on quantization for advances bitnet kernels

What runs where

File	Runtime	Notes
`*.gguf` (ternary)	BitNet.cpp	Primary artifact. Stock llama.cpp will reject it.
Docker image	BitNet.cpp inside the container	linux/amd64. Pull and run; do not rebuild the kernels yourself.
Q8 GGUF, if present	BitNet.cpp or llama.cpp	Compatibility quant. Larger, not the ternary product.

FP16 source weights are not published. They are the seed for later quants.

Quick start

Docker (linux/amd64):

The docker image supports AVX on XEON V2 version with DDR3 RAM . Enjoy

docker run -d \
  --name llama3-ternary-10b \
  -p 7869:7869 \
  --restart unless-stopped \
  cmsmanhattan/llama3-ternary-10b-tq2:latest

JiRack UI

Once the container is running, open your browser and navigate to: http://localhost:7869 This opens the JiRack UI — a clean web interface.
Replace the image name with the tag actually published next to this card.
BitNet.cpp from source (Microsoft repo, not a private fork):

git clone https://github.com/microsoft/BitNet.git
cd BitNet && mkdir build && cd build && cmake .. && cmake --build . -j
./bitnet --model /path/to/Llama3Ternary10b-TQ2_0.gguf -p "Capital of France?"

./llama-completion  -m /mnt/nfs_share/JiRackTernary10_TQ_2/Llama3Ternary10b-TQ2_0.gguf  -c 8192 -t 24 -n 256  -no-cnv -e --no-display-prompt  --temp 0 --repeat-penalty 1.0   -p "<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\nHello<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"  2>/dev/null

# Llama3 Ternary 10b (TQ2_0) — llama-server + web UI / OpenAI API proxy
#
# Put this file in /mnt/nfs_share/Llama3Ternary10b/Docker next to the Dockerfile,
# proxy.py, start.sh, web/ and Llama3Ternary10b-TQ2_0.gguf, then:
#   docker rm -f jirack10b          # once: remove the container started by run.sh
#   docker compose up -d --build    # build + start
#   docker compose logs -f          # wait for "server adds 1 BOS -> ... total 2"
#   docker compose down             # stop and remove
# Settings can be overridden in a .env file next to this one (see .env.example).
# The docker image supports AVX on XEON V2 version with DDR3 RAM . Enjoy  

services:
  Llama3Ternary10b:
    image: cmsmanhattan/llama3-ternary-10b-tq2:latest
    container_name: Llama3Ternary10b
    restart: unless-stopped
    ports:
      - "${HOST_PORT:-7869}:7869"               # web UI + /v1/chat/completions
    environment:
      MODEL_NAME: ${MODEL_NAME:-jirack-10b-ternary}
      THREADS: ${THREADS:-24}
      THREADS_BATCH: ${THREADS_BATCH:-24}
      CTX_SIZE: ${CTX_SIZE:-4096}
      PARALLEL: ${PARALLEL:-1}
      # generation: greedy + no repetition penalty = verified best for this model
      TEMP: ${TEMP:-0}
      TOP_P: ${TOP_P:-0.9}
      REPEAT_PENALTY: ${REPEAT_PENALTY:-1.0}
      REPEAT_LAST_N: ${REPEAT_LAST_N:-64}
      MAX_TOKENS: ${MAX_TOKENS:-512}
      FORCE_SAMPLING: ${FORCE_SAMPLING:-1}     # 1 = ignore temperature etc. sent by clients
      HISTORY_TURNS: ${HISTORY_TURNS:-0}       # 0 = single-turn, like training
      SYSTEM_PROMPT_FILE: ${SYSTEM_PROMPT_FILE:-}   # e.g. /app/system_prompt.txt
    # Optional: edit the system prompt without rebuilding the image
    #volumes:
    #  - ./system_prompt.txt:/app/system_prompt.txt:ro
    healthcheck:
      test: ["CMD-SHELL", "curl -sf http://127.0.0.1:7869/health || exit 1"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 120s
    logging:
      driver: json-file
      options:
        max-size: "20m"
        max-file: "3"

Pin the BitNet.cpp commit you tested. A newer upstream build can break a custom GGUF.

Specs


Base	meta-llama/Meta-Llama-3-8B
Params	~10B after widened FFN
Quant	1.58-bit ternary
Runtime	BitNet.cpp
Context	same as the Llama 3 8B checkpoint used as base, unless the GGUF header says otherwise

Hidden size, intermediate size, and layer count belong in config.json next to the weights. Do not duplicate stale numbers from the 1.5B card here.

Training

Post-quantization QAT / SFT on instruction data (OpenAssistant, then a small SlimOrca slice, stopped early when factual recall slipped). A later mix of SlimOrca with a fact-QA set (GammaCorpus Fact QA and/or Wikipedia QA) is the planned fix for capital / geography drift. This card should be updated with the checkpoint hash after that mix.

Do not treat chat samples below as a benchmark. Replace them with outputs from the checkpoint you actually upload.

JiRack Codding Agent IDE

It is Agent Coding IDE for JiRack Models to run via Ollama on home PC
It good choose for Agent Coding IDE such as Cursor , Windsurf IDE or Devin IDE etc but more safe that ask you to apply changes and review.
Web site https://www.jirack.com
Eclise Plugin site https://marketplace.eclipse.org/content/jirack-coding-agent
Jetbrain Plugin site https://plugins.jetbrains.com/plugin/35073-jirack-agent
VS Code Plugin site https://marketplace.visualstudio.com/items?itemName=JiRackautomationCodingAgentIDE.jirack-agent

JiRack Codding Agent IDE files

Eclise Final release version https://huggingface.co/CMSManhattan/JiRackDeltaNet_27b/resolve/main/jirack_ide_final.zip
Microsoft VS Code plugin https://huggingface.co/spaces/CMSManhattan/README/blob/main/jirack-agent-0.9.0.vsix
Jetbrain plugin https://huggingface.co/spaces/CMSManhattan/README/resolve/main/jirack-jetbrains-0.11.0.zip
JiRack DeskTop UI https://huggingface.co/spaces/CMSManhattan/README/resolve/main/windows_v11.zip

License

Weights and any Llama-derived files: Llama 3 Community License.

Name of this model starts with Llama.
“Built with Llama” is shown on this page.
Commercial use is allowed under that license. The 700M monthly-active-user clause does not apply to a normal product.
You may not relicense the weights under a proprietary CMS Manhattan license.

CMS Manhattan JiRack License, if present, covers only non-Llama materials in this repo (Docker packaging, UI, support). It does not override Meta’s license on the weights.

Acceptable Use Policy: https://www.llama.com/llama3/use-policy/

Commercial work

Ternary QAT on your dataset, replay mix to limit catastrophic forgetting, BitNet.cpp I2_S /AVX / AVX2 / AVX-512 packing, and domain SFT are paid. Weights in this repo are the public checkpoint.

Contact: grabko@cmsmanhattan.com

Not in this repo

Training scripts, lambda-quant recipes, and the FFN-expansion procedure are not part of the public Llama release. The Qwen ternary pipeline published separately is Apache-2.0 and is not a license to ignore the Llama terms on these weights.

Downloads last month: 2,293

Safetensors

Model size

10B params

Tensor type

F16

Model tree for CMSManhattan/Llama3Ternary_10b

Base model

meta-llama/Meta-Llama-3-8B

Quantized

(286)

this model

Quantizations

1 model