Instructions to use Genrius-AI/Qwen3-4B-Pingala8-2bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Genrius-AI/Qwen3-4B-Pingala8-2bit with Trellis:

# No code snippets available yet for this library.

# To use this model, check the repository files and the library's documentation.

# Want to help? PRs adding snippets are welcome at:
# https://github.com/huggingface/huggingface.js

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use Genrius-AI/Qwen3-4B-Pingala8-2bit with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Genrius-AI/Qwen3-4B-Pingala8-2bit
# Run inference directly in the terminal:
llama cli -hf Genrius-AI/Qwen3-4B-Pingala8-2bit

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Genrius-AI/Qwen3-4B-Pingala8-2bit
# Run inference directly in the terminal:
llama cli -hf Genrius-AI/Qwen3-4B-Pingala8-2bit

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Genrius-AI/Qwen3-4B-Pingala8-2bit
# Run inference directly in the terminal:
./llama-cli -hf Genrius-AI/Qwen3-4B-Pingala8-2bit

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Genrius-AI/Qwen3-4B-Pingala8-2bit
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Genrius-AI/Qwen3-4B-Pingala8-2bit

Use Docker

docker model run hf.co/Genrius-AI/Qwen3-4B-Pingala8-2bit

LM Studio
Jan

vLLM

How to use Genrius-AI/Qwen3-4B-Pingala8-2bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Genrius-AI/Qwen3-4B-Pingala8-2bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Genrius-AI/Qwen3-4B-Pingala8-2bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Genrius-AI/Qwen3-4B-Pingala8-2bit

Ollama
How to use Genrius-AI/Qwen3-4B-Pingala8-2bit with Ollama:
```
ollama run hf.co/Genrius-AI/Qwen3-4B-Pingala8-2bit
```
Unsloth Desktop

How to use Genrius-AI/Qwen3-4B-Pingala8-2bit with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Genrius-AI/Qwen3-4B-Pingala8-2bit

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Genrius-AI/Qwen3-4B-Pingala8-2bit"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Genrius-AI/Qwen3-4B-Pingala8-2bit with Docker Model Runner:
```
docker model run hf.co/Genrius-AI/Qwen3-4B-Pingala8-2bit
```

Lemonade

How to use Genrius-AI/Qwen3-4B-Pingala8-2bit with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Genrius-AI/Qwen3-4B-Pingala8-2bit

Run and chat with the model

lemonade run user.Qwen3-4B-Pingala8-2bit-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use Genrius-AI/Qwen3-4B-Pingala8-2bit with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Genrius-AI/Qwen3-4B-Pingala8-2bit

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Genrius-AI/Qwen3-4B-Pingala8-2bit

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Genrius-AI/Qwen3-4B-Pingala8-2bit with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Genrius-AI/Qwen3-4B-Pingala8-2bit

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Genrius-AI/Qwen3-4B-Pingala8-2bit" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3-4B Pingala-8 2-bit (B4)

Author: Harshit Goyal, Genrius AI (harshit@genrius.in). Base model: Qwen/Qwen3-4B (Apache-2.0). Paper: Small-State Coset Trellis Coding at IQ2_XXS Size: An Equal-Bytes Study in llama.cpp (preprint, v2.4), DOI 10.5281/zenodo.23481757. Code and runtime patch: https://github.com/Genrius-AI/pingala8-llm

Try it in one click (Windows, NVIDIA RTX 30/40/50): download pingala8-windows-cuda12.zip from the v2.4-oneclick release, unzip it and double-click run_qwen3_4b.bat. The model downloads once, then a chat page opens in your browser.

This file needs the patched llama.cpp runtime from the code repository. Stock llama.cpp cannot load it.

What this is

B4 is a 2-bit-class quantization of Qwen3-4B at no more bytes than llama.cpp's IQ2_XXS file of the same model. It stores Ungerboeck's 8-state set-partitioned trellis code over 4-weight symbols in a GGUF file, behind a random Hadamard rotation and QuIP's LDLQ rounding. B4 gives each tensor one rate class and sets its input-column scales by minimising an incoherence bound (paper, Section 3). Coding settings (paper, Table 19): 8 states with Ungerboeck labels and a state reset every 32 symbols, a 256-entry int8 4-vector codebook, a random-sign Walsh-Hadamard transform over 256 columns, one fp16 scale per 256 weights per row. On Qwen3-4B, B4 is post hoc: it is the last of a chain of builds evaluated on WikiText-2 during development; only the C4 text is held out.

Files

file	bytes	sha256
`4B-csq-vr3-00001-of-00002.gguf`	995,833,472	`4611812769418b2d0ffb8659f05892d1e3f9c4d20d80b121e93607494ab44243`
`4B-csq-vr3-00002-of-00002.gguf`	247,320,320	`ded47243d30315e49836bf51d652c56fe3b21a288c04f8842d93ff3a70300ff8`

Total 1,243,153,792 bytes (IQ2_XXS of the same model: 1,246,621,312 bytes). Pass the first shard to llama.cpp; it finds the second.

Results (copied from the paper)

WikiText-2 test, context 4096, 73 windows; C4: held-out text, 73 windows; KL divergence: 72 chunks against Q8_0 (Table 1, page 5). Paired difference: B4 minus the row, nats per token, 95% interval (windows on which B4 is better); c: interval corrected for lag-1 autocorrelation.

format	file bytes	PPL	ratio to fp16	paired, B4 - row	C4 PPL	KL div.
fp16	8.05 GB	12.4567	1.000	-	-	-
IQ2_XXS (llama.cpp)	1,246,621,312	28.6083	2.297	-0.542 +/- 0.020 (73/73)	37.28	1.397 +/- 0.012
IQ2_KT, tuned (ik_llama.cpp)	1,245,675,136	18.2721	1.467	-0.094 +/- 0.010c (71/73)	27.33	0.852 +/- 0.010
IQ2_KT, C4-calibrated (ik_llama.cpp)	1,245,914,752	18.1480	1.457	-0.087 +/- 0.011c (71/73)	27.15	0.893 +/- 0.010
B4 (this file)	1,243,153,792	16.6413	1.336	0	24.36	0.585 +/- 0.008
Q2_K (llama.cpp, larger file)	1,669,500,032	15.0133	1.205	-	-	-

Zero-shot accuracy, % (Table 3, page 6; 0-shot continuation scores from llama-perplexity):

format	HellaSwag	WinoGrande	ARC-Easy	ARC-Challenge	MMLU
Q8_0	65.8	65.51	66.14	45.15	38.74
B4 (this file)	57.7	63.77	60.88	37.79	34.01
IQ2_KT, C4-calibrated	53.3	57.93	53.33	35.12	32.91 (e)
IQ2_XXS	47.8	57.46	43.86	29.10	31.07

(e) Rerun with --no-escape (ik_llama.cpp had read escape sequences in the binary task file). Standard errors of a typical 2-bit file: +/-3.1, +/-1.37, +/-2.06, +/-2.79, +/-0.40.

Speed, same-round ratio to IQ2_XXS on one RTX 5070 Ti Laptop GPU (Table 4, page 6): batch-1 decoding (tg128) 1.096 [1.034, 1.158]x, prompt processing (pp512) 0.741 [0.710, 0.772]x, batched decoding of 2 / 4 / 8 sequences 0.18 / 0.26 / 0.34x.

How to run

Build llama.cpp at commit 7ac59a6e3ad851cd41af00f678effab0598ba9a8 with the CUDA patch from the code repository:

git clone https://github.com/Genrius-AI/pingala8-llm
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout 7ac59a6e3ad851cd41af00f678effab0598ba9a8
git apply ../pingala8-llm/patches/llama_cuda_vs_7ac59a6.patch
cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120 -DCMAKE_BUILD_TYPE=Release -DLLAMA_OPENSSL=OFF
cmake --build build -j8 --target llama-cli llama-perplexity llama-bench

Then (all layers on the GPU):

hf download Genrius-AI/Qwen3-4B-Pingala8-2bit --local-dir qwen3-4b-pingala8
export GGML_F44=4 GGML_F44_BATCH=2 GGML_VR_NATIVE=1 GGML_VRC_V=5 GGML_VRC_NWP=8 GGML_VR_RL=1 GGML_VR_PRMG=1 GGML_VR_PPC=2
./build/bin/llama-cli -m qwen3-4b-pingala8/4B-csq-vr3-00001-of-00002.gguf -ngl 99 -p "The trellis code"
./build/bin/llama-perplexity -m qwen3-4b-pingala8/4B-csq-vr3-00001-of-00002.gguf -f wikitext2_test_hfjoin.raw -c 2049 --ppl-stride 4095 -ngl 99

The perplexity command is the paper's protocol (expected final value [73]16.6413); the code repository rebuilds the text file and documents the build (B4.md). The llama-cli line is a usage example and was not part of the measured runs. Set CMAKE_CUDA_ARCHITECTURES for your GPU; on WSL2 see bin/cuda_env.sh in the code repository.

Limitations

CUDA only on the measured path, measured on one laptop GPU (RTX 5070 Ti Laptop, WSL2); no Metal or Vulkan.
Prompt processing is slower than IQ2_XXS (0.74x) and batched decoding of 2 to 8 sequences is much slower (0.18-0.34x). The batch-1 decoding edge is not treated as established (paper, Section 4.2).
At 2 bits the model is clearly worse than the full-precision model (perplexity 1.336x fp16).
With the same recipe, B4 loses to IQ2_KT on Qwen3-14B (paper, Section 4.3); results do not transfer to every model.
Refusal, bias and hallucination behaviour after quantization were not evaluated. Evaluate the model for your own use.

Citation

@misc{goyal2026pingala8,
  title        = {Small-State Coset Trellis Coding at {IQ2\_XXS} Size: An Equal-Bytes Study in {llama.cpp}},
  author       = {Goyal, Harshit},
  year         = {2026},
  month        = oct,
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.23481757},
  url          = {https://doi.org/10.5281/zenodo.23481757},
  note         = {Preprint, version 2.4. Code: https://github.com/Genrius-AI/pingala8-llm}
}

License and notice

Apache-2.0, inherited from Qwen3-4B (Copyright 2024 Alibaba Cloud); see LICENSE and NOTICE. The weights are a quantized derivative of Qwen3-4B made by Genrius AI. "Pingala-8" and "Genrius AI" are names of Genrius AI; the license grants no trademark rights.

Downloads last month: 89

GGUF

Model size

1B params

Architecture

qwen3

Hardware compatibility

We're not able to determine the quantization variants.

View all variants

Model tree for Genrius-AI/Qwen3-4B-Pingala8-2bit

Base model

Qwen/Qwen3-4B-Base

Finetuned

Qwen/Qwen3-4B

Quantized

(359)

this model