aya-23-8b-nvfp4-mixed

CohereLabs/aya-23-8B quantized to NVFP4 via modelopt, calibrated on mixed Arabic dialogue. Weights and activations are quantized; this is a real packed checkpoint, not fake-quant.

Hardware: needs sm_100+ (B200, GB200, RTX 50) to run — an H100 can export this but cannot load it.

Calibration

set	calib3_mixed.txt (mixed)
dataset	Almheiri/ArabCulture-Dialogue
configs / split	KSA, UAE / eval
dataset revision	9acd60cbbb4fab834b1054e9035499f035c5c0af
sampling seed	1448
samples	128 dialogues, max 512 tokens
file digest	sha256[:16] = 4ad8c6905f924903

Calibration is forward passes with observers attached — no labels, no gradients, no generation. Observers record per-tensor absmax as data flows through. Weight scales are computed data-free from the weights themselves, so only the 224 activation ranges depend on this set.

Three sibling checkpoints exist per model and format — -msa, -gulf and -mixed. They differ only in calibration register: calib1_msa and calib2_gulf are line-parallel (the same 128 dialogues in Modern Standard Arabic and in Gulf dialect), and calib3_mixed is 128 distinct dialogues drawn from both. Any behavioural difference between them is attributable to the calibration register and nothing else.

Quantization

format	NVFP4
backend	modelopt
weights	quantized
activations	quantized, 224 calibrated tensors
mean weight MSE, |W - Q(W)|	3.676e-06
weight quantizers	224
size	16.06 GB -> 6.04 GB (2.66x smaller)
activation amax, min	1.614
activation amax, median	7.750
activation amax, max	440.2 (model.layers.1.mlp.down_proj)
max / median	57x

A large max/median amax ratio means a few tensors carry outliers far outside the typical range; those tensors dominate the activation quantization error.

Not benchmarked here

This checkpoint was produced by a build pipeline that computes no accuracy, perplexity, KL or agreement metrics. Do not read the numbers above as quality scores — they describe the quantization, not the model's behaviour. Evaluation lives in the ArabicFP4 benchmark.

Note also that this is a W(4/8)A(4/8) checkpoint. The published ArabicFP4 benchmark measured weight-only simulated quantization with fp16 activations, so its accuracy deltas are a floor for what this costs, not a prediction.