Instructions to use WiTi4Bit/gemma-3-1b-it-qnm-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use WiTi4Bit/gemma-3-1b-it-qnm-4bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="WiTi4Bit/gemma-3-1b-it-qnm-4bit")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("WiTi4Bit/gemma-3-1b-it-qnm-4bit")
model = AutoModelForCausalLM.from_pretrained("WiTi4Bit/gemma-3-1b-it-qnm-4bit", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use WiTi4Bit/gemma-3-1b-it-qnm-4bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "WiTi4Bit/gemma-3-1b-it-qnm-4bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WiTi4Bit/gemma-3-1b-it-qnm-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/WiTi4Bit/gemma-3-1b-it-qnm-4bit

SGLang

How to use WiTi4Bit/gemma-3-1b-it-qnm-4bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "WiTi4Bit/gemma-3-1b-it-qnm-4bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WiTi4Bit/gemma-3-1b-it-qnm-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "WiTi4Bit/gemma-3-1b-it-qnm-4bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WiTi4Bit/gemma-3-1b-it-qnm-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use WiTi4Bit/gemma-3-1b-it-qnm-4bit with Docker Model Runner:
```
docker model run hf.co/WiTi4Bit/gemma-3-1b-it-qnm-4bit
```

You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Quantized Gemma 3 Model Card

Model Details

Base Model: google/gemma-3-1b-it
Model Type: Gemma3ForCausalLM (model_type=gemma3_text)
Quantization Format: Sub-block Q4M (U8 Containers)
Quantization Date: November 3, 2026
Research Publication: Johandri, A. (2026). Precision Transition Dynamics: Standardizing BF16 to U8 Containers in Sub-Block Quantization for Edge LLM Inference. Zenodo. https://doi.org/10.5281/zenodo.23921551

Overview

This repository contains a quantized version of Gemma 3 (1B Instruction-Tuned) optimized for efficient edge LLM inference using a customized Q4M sub-block quantization toolchain.

The quantization pipeline transitions the original BF16 weights (approximately 2.00 GB) down to a highly compressed format of ~584.8 MB for model.safetensors, achieving a 3.42x reduction (70.8% smaller) while maintaining critical precision layers.

Quantization Configuration & Specifications

Block Structure: Q4M format with 32 elements per block (consisting of a 2-byte FP16 scale + 16 bytes of 4-bit packed data -> 18 bytes/block).
Mantissa & Rounding: 4-bit mantissa (min(bits, 10)) utilizing round-half-up rounding with rounding nearest (seed=42).
Precision Transition Threshold: Float tensors below 1,000,000 elements are stored via split-FP16 to preserve accuracy, while tensors >= 1,000,000 elements undergo QnM compression.
Tensor Distribution:
- 131 QnM Tensors (>= 1M elements): Comprising ~98.5% of total weight bytes (1.97 GB -> 553.7 MB), including model.embed_tokens.weight (603.98 MB).
- 209 Split-FP16 Tensors (< 1M elements): Comprising ~1.5% of weight bytes, keeping LayerNorms, k_proj, and v_proj in native 16-bit precision.

File Size Comparison

Component	Original (BF16)	Quantized (Q4M U8)	Reduction
`model.safetensors`	2.00 GB (1.86 GiB)	584.8 MB (0.545 GiB)	-70.8% (3.42x)
Total Model Directory	~2.04 GB	~624 MB (excl. cache)	Significant footprint reduction

Metadata Tracking

The output model.safetensors header embeds full traceability configurations under the qnm.* namespace:

qnm.format = qnm-packed-safetensors
qnm.bits = 4 
qnm.blok = 32 
qnm.mantisa = 4 
qnm.mode = signed-grid
qnm.block_bytes = 18 
qnm.ambang_f16 = 1000000
qnm.precision_transition = eksponen{3,6}|mantisa{0,9}{1,8}{2,7}{4,5}
qnm.pembulatan = round-half-up

Verification & Validation

Self-Test Suite (--uji): Passed round-trip verification, error bounds, scaling, and symmetry checks across both 32-element and 8-element block variants.
Round-Half-Up Validation: Passed utilizing all 10 mantissa bits with proper carry propagation and saturation limits (0x7BFF).

Documentation, Download Links & Repository Access

You can access the formal research publication, source code, runtime scripts, and model weights through the following official links provided by the POKDARTIK AI Engineering Team.

1. Research Paper / Journal (Zenodo DOI)

DOI URL: https://doi.org/10.5281/zenodo.23921551

2. Source Code & Runtime Repository (GitHub)

Repository URL: https://github.com/WiTi4Bit/runtime_chat

Clone Command:

git clone https://github.com/WiTi4Bit/runtime_chat.git

3. Quantized Model Weights (Hugging Face)

Model Repository: https://huggingface.co/WiTi4Bit/gemma-3-1b-it-qnm-4bit

Clone Command (Git LFS required):

git lfs install
git clone https://huggingface.co/WiTi4Bit/gemma-3-1b-it-qnm-4bit

Quick Setup Guide

Clone both repositories into your local workspace.

Install the required Python dependencies:

pip install torch transformers safetensors

Run the chat runtime script:

python3 runtime_chat/chat_runtime_public.py ./gemma-3-1b-it-qnm-4bit

License & Terms of Use

This software and model are intended exclusively for non-commercial, educational, research, and peer-review purposes.

Users are permitted to download, run, and distribute this model provided that proper attribution is given to POKDARTIK (https://pokdartik.or.id) and the accompanying Zenodo publication.
Strict prohibition against utilizing this model or conversion scheme for any commercial purposes.

Downloads last month: 8

Safetensors

Model size

0.6B params

Tensor type

Model tree for WiTi4Bit/gemma-3-1b-it-qnm-4bit

Base model

google/gemma-3-1b-pt

Finetuned

google/gemma-3-1b-it

Finetuned

(571)

this model