Goetia-26B-A4B-v1.4-FP8-Dynamic

FP8 Dynamic (W8A8, FP8_DYNAMIC) quantization of Naphula/Goetia-26B-A4B-v1.4.

Source

• Original model: Naphula/Goetia-26B-A4B-v1.4 (Gemma-4-26B-A4B MoE, BF16), a roleplay finetune.
• License: apache-2.0, inherited from the original model.

How it was made

llm-compressor, data-free one-shot, with the exact same recipe as the production quant cloud19/G4-MeroMero-26B-FP8-Dynamic-Uncensored:

default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore: [lm_head, 're:.*embed.*', 're:.*router', 're:.*vision_tower.*']
      scheme: FP8_DYNAMIC
      bypass_divisibility_checks: false

Weights are FP8 per-channel (static), activations FP8 per-token (dynamic). Kept in BF16: embeddings, lm_head, MoE routers and the vision tower.

Weights size: 26.67 GiB (BF16 original ~48.4 GiB). Fits a single 96 GB GPU with ample room for KV cache.

Caveat: vision tower

The source checkpoint is missing 4 of 359 vision-tower tensors; those tensors are randomly initialized in this repository. This has no effect on text generation, but the model should be served text-only (see --limit-mm-per-prompt below).

Serving with vLLM

vllm serve cloud19/Goetia-26B-A4B-v1.4-FP8-Dynamic \
  --served-model-name goetia \
  --quantization compressed-tensors \
  --kv-cache-dtype fp8 \
  --max-model-len 16384 \
  --max-num-seqs 384 \
  --limit-mm-per-prompt '{"image": 0}' \
  --no-enable-chunked-prefill