Altworld/Astrea-R8-Chat-9B-FP8

Altworld Astrea R8 Chat 9B — FP8

This is the portable FP8 release of Altworld/Astrea-R8-Chat-9B, Altworld's open 9B creative-writing and chat model.

Try Astrea · API · Documentation

The checkpoint uses LLM Compressor's compressed-tensors format and can be served directly by compatible versions of vLLM. It contains six safetensor shards totalling 11.09 GiB, compared with 17.53 GiB for the BF16 weights.

Quantization

The checkpoint was generated from the released merged BF16 model with:

• LLM Compressor 0.12.0;
• compressed-tensors 0.17.1;
• FP8_DYNAMIC, using E4M3 weights and BF16 per-output-channel scales;
• dynamic per-token FP8 activation quantization at inference time;
• data-free round-to-nearest post-training quantization.

All 248 two-dimensional language-model linear weights are stored in FP8. The following sensitive or non-linear components remain in BF16:

• input token embeddings;
• lm_head;
• Qwen3.5 linear-attention convolution kernels;
• the complete vision tower.

No calibration or training dataset was used during conversion.

Running with vLLM

Use a recent vLLM release with compressed-tensors support and an accelerator with supported FP8 kernels:

vllm serve Altworld/Astrea-R8-Chat-9B-FP8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90

The quantization configuration is embedded in config.json; no separate --quantization argument should be necessary.

Recommended sampling for writing:

{
  "temperature": 0.8,
  "min_p": 0.025,
  "repetition_penalty": 1.08
}

Use temperature: 0.2 for ordinary factual chat. The bundled chat template works without a system prompt and defaults to hidden thinking.

Validation

Before upload, this artifact passed the following local checks:

• every original tensor is present with the expected shape;
• all 248 targeted weights are E4M3 with matching BF16 channel scales;
• all excluded components retain their original BF16 dtype;
• the sharded safetensor index exactly matches all 1,008 stored tensors;
• AutoConfig, AutoTokenizer, and AutoProcessor load successfully;
• Qwen3_5ForConditionalGeneration loads the compressed checkpoint and completes a text-generation smoke test.

The local Transformers smoke test dequantizes weights for its CPU fallback and therefore does not measure FP8 serving speed. Quantization changes model numerics, so applications should evaluate this checkpoint on their own prompts before replacing the BF16 release. Quant-specific Altworldbench results are not claimed on this card.

Model quality and limitations

See the BF16 model card for training lineage, benchmark methodology, samples, usage guidance, and limitations. In particular:

• Astrea is text-first; inherited vision behavior was not part of its release evaluation;
• English dominates training and evaluation;
• outputs can be confidently wrong and must not be treated as authoritative in high-stakes settings.

License

Apache-2.0, matching the source model. See LICENSE and NOTICE.