DZAIR-FP16

Half-precision build of algerian-nlp/DZAIR, the 105.3M-parameter encoder for Algerian Darija. Same weights, half the bytes, same outputs to five decimal places.

Fidelity

Measured by make quantize-fp16 REPO=DZAIR on a fixed 4×96 token probe with one padded row, against the fp32 release. The full report ships as quant_report.json.

	fp32	fp16
Directory size	422.3 MB	211.7 MB (2.00× smaller)
model.safetensors	421.2 MB	210.6 MB
Cosine against fp32	1.000000 (reference)	0.999999
Max absolute difference	0.000000	0.007960

Release gate: cosine ≥ 0.999.

This build was broken until 2026-09-19 and is worth re-downloading. The earlier upload shipped an fp32 config.json alongside half-precision weights, so transformers reloaded them as float32 — the file was half the size but bought nothing at inference. Loading it genuinely in float16 returned an all-zero hidden state, because RMSNorm squared its input in half precision: trunk activations reach about 316, and 316² = 99,856 overflows the float16 maximum of 65,504, so the mean became inf and its reciprocal square root became 0. The norm statistic is now computed in float32 and cast back — float32 outputs are bit-identical to before (verified: max absolute difference exactly 0.0), and float16 works.

Usage

import torch
from transformers import AutoModel, AutoTokenizer

REPO = "algerian-nlp/DZAIR-FP16"
tokenizer = AutoTokenizer.from_pretrained(REPO, trust_remote_code=True)
encoder = AutoModel.from_pretrained(REPO, trust_remote_code=True).eval()

texts = ["يعطيك الصحة خويا", "ya3tik saha khoya, bon courage f projet"]
inputs = tokenizer([t.lower() for t in texts], padding=True, return_tensors="pt")
with torch.inference_mode():
    hidden = encoder(**inputs).last_hidden_state
print(hidden.dtype, hidden.shape)  # torch.float16 torch.Size([2, N, 768])

Lowercase Latin spans before encoding — the vocabulary was built over lowercased Latin and raw uppercase costs about 10% fertility. Arabizi phoneme digits (3, 7, 9) are atomic pieces; never transliterate them away.

Files

file	size	contents
model.safetensors	210.6 MB	float16 weights
config.json	1 KB	architecture, dtype: float16, auto_map
modeling_dzair.py	59 KB	the architecture in one self-contained file
tokenizer.model, tokenizer_config.json	about 1.0 MB	48k SentencePiece Unigram via DebertaV2Tokenizer, specials at ids 0–4, right padding
tokenizer_rules.yaml	2 KB	versioned normalisation rules
quant_report.json	under 1 KB	the measurement above

Licence

Apache-2.0, inherited from the base model. Read the licence composition on the base card before redistributing derivatives: a permissive grant on the weights makes no claim about the underlying text, most of which has no resolvable licence.