Yusiko/khazri-3

Khazri 3

An Azerbaijani-focused foundation for multilingual AI.

1B parameters · 20B training tokens · 4 target languages · Apache 2.0

Khazri 3 brings Azerbaijani into the heart of a compact, multilingual language model. Trained from scratch, it offers an open foundation for developers, researchers, and teams building language products for Azerbaijani-speaking users.

This release shares the final 20,000,094,278-token pretraining checkpoint, with 1,002,522,624 parameters and a training focus spanning Azerbaijani, English, Russian, and Turkish.

Built for your next language product

• Azerbaijani at the center. A multilingual training direction that includes Azerbaijani alongside English, Russian, and Turkish.
• An independent foundation. Weights trained from scratch with a dedicated 49,152-token byte-level BPE tokenizer and a Llama-compatible architecture.
• A focus on data quality. The team reports using cleaned, deduplicated web text and material from approximately 1,000 books in its training corpus.
• Compact and accessible. Approximately 1B parameters, standard Transformers support, and safetensors weights. The usage example supports FP16 on CUDA devices such as a T4 and FP32 on CPU.
• Open for adaptation. Apache 2.0 licensing makes this foundation available for experimentation, fine-tuning, and commercial integration under the license terms.

Use Khazri 3 as a starting point for domain-specific language models, Azerbaijani text products, and custom assistants after suitable instruction tuning. This is a BASE model: instruction following, web search, and tool calling are future adaptation goals, not capabilities already trained into this checkpoint.

Benchmark snapshot

The team-provided comparison graphic reports 28.6% MMLU and 55.0% HellaSwag for Khazri-3 1B. The complete values from that graphic are reproduced below.

Model	MMLU accuracy (%)	HellaSwag accuracy (%)
Khazri-3 1B	28.6	55.0
TinyLlama 1.1B	26.0	59.2
Pythia 1B	25.0	47.2
Qwen3 0.8B	29.0	45.7
Llama 3.2 1B	39.5	61.1
Gemma 3 1B	42.0	62.3

Source: comparison graphic supplied by the Khazri team, dated October 1, 2026. These are reported reference figures, not an independently reproduced evaluation. The evaluated checkpoint, shot counts, dataset splits, sample counts, precision, and scoring method are not specified in the graphic. These figures are therefore not verified results for the final 20B-token checkpoint, and differences across rows may reflect different evaluation protocols.

Start building

Load the model directly from Yusiko/khazri-3. No custom remote model code is required. Start with text completion, then fine-tune for your own use case.

Model details

Property	Value
Architecture	Llama-compatible causal decoder
Parameters	1,002,522,624
Layers	20
Hidden width	2,048
Feed-forward width	5,632
Attention heads / KV heads	16 / 4
Vocabulary	49,152
Trained context	2,048 tokens
RoPE theta	10,000
Input/output embeddings	Tied
Training tokens	20,000,094,278
Final checkpoint	step-000109896-c2123f2d
Format	Hugging Face Transformers; sharded safetensors
BOS / EOS / PAD IDs	2 / 3 / 0

The training phase targets Azerbaijani, English, Russian, and Turkish. This BASE checkpoint has not undergone instruction, chat, or tool-calling SFT. Generated continuations may contain factual errors or repetition; evaluate the model in your intended language and domain before deployment.

Use with Transformers

Known working library versions: transformers==4.57.1, tokenizers==0.22.2, and safetensors==0.6.2. Install a PyTorch build appropriate for your device.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "Yusiko/khazri-3"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if device == "cuda" else torch.float32

tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=False)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    torch_dtype=dtype,
    use_safetensors=True,
    trust_remote_code=False,
    attn_implementation="sdpa",
).to(device).eval()

prompt = "Sual: Azərbaycanın paytaxtı hansı şəhərdir?\nCavab:"
ids = [2] + tokenizer.encode(prompt, add_special_tokens=False)
max_new_tokens = 64
assert len(ids) + max_new_tokens <= 2048
inputs = torch.tensor([ids], device=device)

torch.manual_seed(42)
with torch.inference_mode():
    output = model.generate(
        input_ids=inputs,
        attention_mask=torch.ones_like(inputs),
        max_new_tokens=max_new_tokens,
        do_sample=True,
        temperature=0.9,
        top_p=0.95,
        bos_token_id=2,
        eos_token_id=3,
        pad_token_id=0,
    )

print(tokenizer.decode(output[0, len(ids):], skip_special_tokens=True))

For greedy decoding, use do_sample=False and omit temperature and top_p. Prompt and generated tokens together must fit the 2,048-token context. A T4 can use FP16; CPU inference uses FP32 in the example.

Files and provenance

The repository includes all eight model shards, their index, tokenizer files, model configuration, and export/training metadata. This is a parameters-only inference export. Optimizer state and training data are not included.

• Tokenizer SHA-256: 04cdc8532536cd629f4114e91e9a7dcc60594010ebcea578e3eb2c623766c3d9
• Source checkpoint manifest SHA-256: c3207ca9e60fb91b8c0ccc4c660ec89fafe329f3a9bfe8b66d08f95792306690
• Export manifest SHA-256: 18f4424a22524c38dbbc505465364d5c51f0fb7c959ae2559cf552c266ff3c79

The export manifest hashes the original model/configuration files. This model card was added for publication and is not listed in that export manifest.

License

Apache-2.0, as selected for this repository.