Weidows/laya-multilingual-GGUF

laya-multilingual GGUF

GGUF conversion of convaiinnovations/laya-multilingual (Apache-2.0), encoder only (mmBERT-base, 22 layers, hidden 768). Bit-identical to an independent community conversion (134/134 tensors, max diff 0).

The decision head (36 tensors, 30 MB) ships separately as laya-multilingual-head.safetensors (same split as an mmproj file); laya_head.py has load_head(path) returning config + weights under original PyTorch names.

Token ids must come from the shipped HF tokenizer.json (Gemma-style tokenizer; llama.cpp text tokenization not reliable for this vocab).

Files

file	size	worst rel err vs F16 (dequant)
laya-multilingual-F16.gguf	629 MB	-
laya-multilingual-Q8_0.gguf	341 MB	4.2e-03
laya-multilingual-Q6_K.gguf	271 MB	1.6e-02
laya-multilingual-Q5_K_M.gguf	258 MB	3.7e-02
laya-multilingual-Q4_K_M.gguf	249 MB	7.5e-02
laya-multilingual-Q4_K_M-imat.gguf	249 MB	1.3e-01 (worst-tensor rel err; imatrix shifts bits to sensitive weights)
laya-multilingual-Q4_K_S.gguf	242 MB	7.5e-02
laya-multilingual-Q3_K_M.gguf	236 MB	1.9e-01
laya-multilingual-IQ4_XS.gguf	236 MB	8.4e-02
laya-multilingual-IQ4_XS-imat.gguf	235 MB	3.5e-01 (worst-tensor rel err)

imatrix stats from 400-line 10-language Wikipedia sample (imatrix-laya.dat used at quant time; dat file not shipped). Weight-space rel err is a smoke signal only — see eval below.

Eval (end-to-end, llama.cpp server + original decision head)

12 samples (6 XNLI choice across en/zh/ar/hi/es/ja + 3 dept choice + 3 urgency noul), same HF token ids into PyTorch FP32 and each GGUF via llama.cpp server (--pooling none, --embd-normalize -1), identical head weights. F16 matches FP32 exactly (12/12, drift 0.0015).

quant	decision agreement	mean conf drift
F16	12/12 (1.00)	0.0015
Q8_0	12/12 (1.00)	0.0043
Q6_K	11/12 (0.92)	0.0103
Q5_K_M	12/12 (1.00)	0.0309
Q4_K_M	12/12 (1.00)	0.0381
Q4_K_M-imat	12/12 (1.00)	0.0347
Q4_K_S	12/12 (1.00)	0.0407
Q3_K_M	10/12 (0.83)	0.0891
IQ4_XS	12/12 (1.00)	0.0157
IQ4_XS-imat	12/12 (1.00)	0.0167

Mismatches only in Q6_K (1 hi) and Q3_K_M (1 ar + 1 hi, drift 0.34+). Recommendation: Q8_0 for anything important; Q4_K_M-imat / IQ4_XS-imat for size; avoid Q3_K_M on non-English.

Use

llama-server -m laya-multilingual-Q8_0.gguf --embeddings --pooling none -c 2048 -ub 2048 -b 2048 --port 8080
pip install laya requests torch safetensors

import requests, torch, laya
from laya_head import load_head

agent = laya.load("convaiinnovations/laya-multilingual", device="cpu")
_, head = load_head("laya-multilingual-head.safetensors")
sd = agent.model.state_dict()
for k, v in head.items():
    if k in sd:
        sd[k].copy_(v)

D = agent.model.encoder.config.hidden_size
def llamacpp_encoder(input_ids, attention_mask=None, **_):
    out = []
    for i, row in enumerate(input_ids):
        n = int(attention_mask[i].sum())
        r = requests.post("http://localhost:8080/embedding", json={"content": [row[:n].tolist()]}).json()
        h = torch.zeros(input_ids.shape[1], D); h[:n] = torch.tensor(r[0]["embedding"]); out.append(h)
    return torch.stack(out)