AMALIA-9B-0626-DPO-FP8

FP8 quantization of amalia-llm/AMALIA-9B-0626-DPO, the European Portuguese language model, for serving with vLLM.

To the best of our knowledge this is the first FP8 quantization of AMALIA. Every other quantization published on the Hub targets llama.cpp (GGUF) or Apple Silicon (MLX).

Why FP8

The goal was a version that runs well on server GPUs without measurable quality loss. FP8 was chosen over 4-bit formats for one reason that matters specifically for a language-specialised model: FP8 dynamic quantization requires no calibration data. Weight scales are computed per channel and activation scales at runtime.

4-bit schemes need a calibration pass, and the default calibration datasets are English. Calibrating a European Portuguese model on English text would degrade exactly the property that makes the model worth using.

Results

Measured against the BF16 original on an NVIDIA DGX Spark (GB10 Grace-Blackwell, 121 GB unified memory, sm_121a).

Perplexity over 401 tokens of European Portuguese across four registers. The colloquial passage deliberately contains comboio, pequeno-almoço, casa de banho and telemóvel, so that any degradation of specifically European vocabulary would show up.

Register	BF16	FP8	Delta
Literary	15.6962	16.0568	+2.30%
Administrative	4.7820	4.8075	+0.53%
Colloquial	9.8653	10.1136	+2.52%
Technical	18.1637	17.8324	-1.82%
Global	11.0987	11.2033	+0.94%

Serving

	BF16	FP8
Size on disk	18 GB	9.6 GB
Throughput (single stream, DGX Spark)	12.2 tok/s	22.3 tok/s
KV cache at the same memory fraction	140,912 tokens	190,112 tokens
Startup time	270s	180s

Language markers. Across 12 generation prompts, both variants produced 7 European Portuguese lexical markers and zero Brazilian ones, with the European gerund construction (estou a fazer) and never the Brazilian one (estou fazendo).

Caveat

Sub-1% perplexity difference does not mean identical answers. Greedy decoding amplifies tiny logit differences: across 12 prompts, none of the FP8 answers matched the BF16 answer token for token, even though the aggregate quality is equivalent. On one geography question the FP8 answer placed Madeira near the Strait of Gibraltar, which is wrong, where the BF16 answer correctly placed it west of Morocco.

Statistical equivalence is not answer-by-answer equivalence. Use accordingly.

Usage

vllm serve <this-repo> \
  --served-model-name amalia \
  --max-model-len 32768 \
  --host 0.0.0.0 --port 8000

vLLM detects compressed-tensors automatically. No extra flags are needed.

How it was made

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=["lm_head"],
)
oneshot(model=model, recipe=recipe)

Total time: 209 seconds, of which 2 seconds are the quantization itself. The rest is reading the model from disk and writing it back. llm-compressor selects a DataFreePipeline because the recipe needs no calibration data.

lm_head is kept in original precision.

A note on weight-only quantization

We also produced a weight-only variant (W8A16, input_activations=None) expecting it to be more accurate, since activation quantization is usually where the error lives. It was worse: +2.31% perplexity against +0.94% for W8A8.

The configuration files show that weights are quantized identically in both variants, so the entire difference comes from the compute path. W8A8 runs on the Blackwell native FP8 tensor cores with FP32 accumulation; W8A16 upconverts the weights back to BF16. On an accelerator with native FP8 support, using that support turned out to be more accurate than avoiding it.

Em português

Esta é uma quantização FP8 do AMALIA-9B, o modelo de língua de português europeu, para ser servida em vLLM. Foi escolhido FP8 em vez de 4 bits porque não precisa de calibração, e calibrar um modelo de português com textos ingleses degradaria precisamente aquilo que o distingue.

Resultado: metade do tamanho, quase o dobro do débito, e menos de 1% de degradação de perplexidade medida sobre quatro registos de português europeu.

Attribution

Base model: amalia-llm/AMALIA-9B-0626-DPO, Apache 2.0. All credit for the model itself belongs to the AMALIA team. This repository contains only a quantization.

Quantization method: llm-compressor.