attentionlab/bayon-it-ggufs

Bayon Instruct — GGUF quants

This repository contains GGUF conversions and quantizations of attentionlab/bayon-it (Bayon Instruct), a ~109M-parameter Gemma-architecture decoder-only language model for Khmer.

The original FP32 weights live in the upstream repo. To re-quantize yourself, start from the original safetensors files:

• Repo: attentionlab/bayon-it
• Weights: model.safetensors
• Tokenizer: tokenizer.model · tokenizer.vocab

Bayon Instruct is instruction-tuned (LoRA) from attentionlab/bayon, trained with data from attentionlab/bayon-sft. Full details (intended use, evaluation, training recipe, limitations) are in the original model card.

Files

File	Format	Size	Notes
attentionlab_bayon-it-bf16.gguf	BF16	209 MB	Unquantized conversion (16.00 BPW); the source for the quants below
attentionlab_bayon-it-q8_0.gguf	Q8_0 (int8)	111 MB	8.51 BPW, near-lossless
attentionlab_bayon-it-q4_k_m.gguf	Q4_K_M	64 MB	4.85 BPW, with the standard k-quant mixture (token embeddings, attn_v and ffn_down bumped to Q6_K)

All three files come from a single BF16 conversion; the quants are mixed per-tensor types (57 f32 norm tensors are preserved), matching what llama-quantize produces by default.

Usage

Requires a reasonably recent llama.cpp (GGUF V3, gemma architecture; build used: 0.5.0-dev, 2026-10-17).

# Quick test
llama-cli -m attentionlab_bayon-it-q8_0.gguf -p "សួស្តី ពិភពលោក"

# Serve an OpenAI-compatible endpoint
llama-server -m attentionlab_bayon-it-q4_k_m.gguf -c 2048

More general quantization-format tips: llama.cpp/gguf docs.

How these were produced

Converted with llama.cpp (convert_hf_to_gguf.py + llama-quantize, build 0.5.0-dev / commit unknown, built 2026-10-17 via the llama-cpp-python 0.3.36 vendored sources):

python3 convert_hf_to_gguf.py models/attentionlab/bayon-it --outtype bf16 \
        --outfile output/attentionlab_bayon-it-bf16.gguf

llama-quantize output/attentionlab_bayon-it-bf16.gguf output/attentionlab_bayon-it-q8_0.gguf q8_0
llama-quantize output/attentionlab_bayon-it-bf16.gguf output/attentionlab_bayon-it-q4_k_m.gguf q4_k_m

Original model (summary from upstream)

Property	Value
Base model	attentionlab/bayon
Parameters	~109M (with 5,000-token Khmer BPE tokenizer, byte fallback)
Architecture	Decoder-only transformer (GemmaForCausalLM, 28 layers, hidden 512, GQA 8/2 heads, dim. 8192 context)
Pretraining	1.361B Khmer tokens from FineWeb-2
Instruction tuning	18,000 Gemini-distilled Khmer SFT examples, LoRA r=16 α=32
License	Apache-2.0

See the original model card for intended use, benchmark results (Khmer history & culture, language-bleed ratio), training details, limitations, and safety considerations.