smolaya-int8

anon767tom/smolaya with its CPU int8 quantisation already applied: 456 MB instead of 646 MB, about 2.8× faster than laya on CPU, and no quantisation step at load time.

• Encoder attn.Wqkv, attn.Wo and mlp.Wi are stored as int8 weights with per-output-channel scales (symmetric, zero point 0) and run with PyTorch's dynamic int8 kernels.
• mlp.Wo stays in 16-bit on disk and runs in fp32: its inputs carry large activation outliers, and quantising it costs several accuracy points.
• The stored int8 weights are exactly what quantize_int8.py in the smolaya repo produces, so the outputs are identical to quantising smolaya yourself (maximum logit difference 0.0 on 50 test items).

Usage

# pip install transformers torch
from transformers import pipeline

clf = pipeline("zero-shot-classification", model="anon767tom/smolaya-int8", trust_remote_code=True)
clf("The film was a complete waste of two hours.", candidate_labels=["negative", "positive"])

Custom questions, multi_label=True, the full laya questions={...} schema and model.predict(...) work exactly as described in the smolaya card.

The int8 kernels are CPU-only. On a GPU the same weights run dequantized (slower than the fp16 smolaya, so use that one there). This repo loads with transformers only, not with laya.load.

Accuracy and speed

Same 12,398 evaluation items as the smolaya card:

Task	laya	smolaya	smolaya-int8
SST-2	0.918	0.942	0.939
ARC-Easy	0.511	0.602	0.596
BoolQ	0.835	0.851	0.849
MNLI	0.883	0.892	0.891
AG News	0.923	0.929	0.924
Pooled	0.812	0.839	0.836

int8 vs smolaya: −0.4 pt [95% CI −0.6, −0.1]. Mean CPU time per item (batch 1, 4 threads, Ryzen 3 4300U): laya 0.71 s, smolaya-int8 0.26 s.

License and attribution

Apache-2.0. Quantised from anon767tom/smolaya, itself derived from convaiinnovations/laya by Convai Innovations (ModernBERT-large encoder by Answer.AI).