Sophea-Genesis-1-FP8-dynamic

FP8 (W8A8-dynamic) quantization of KIEFERSA/Sophea-Genesis-1, produced with LLM Compressor.

Weights are quantized to float8_e4m3fn per-channel; activations are quantized to FP8 per-token, dynamically, at inference time. No calibration data was used: FP8-dynamic derives weight scales directly from the weights.

• Base model: KIEFERSA/Sophea-Genesis-1 (27B, Qwen3.6 hybrid attention/linear-attention architecture)
• Format: compressed-tensors
• Size: ~29 GB (from ~55 GB bf16)

What is and isn't quantized

Sophea-Genesis-1 is a hybrid model: 64 decoder layers, of which 48 are Mamba-style linear-attention layers and 16 are full self-attention, plus a vision tower. To preserve accuracy, the following are kept in bfloat16 and excluded from quantization:

• lm_head
• linear_attn.in_proj_a / linear_attn.in_proj_b in every linear-attention layer: these produce the SSM's decay/step control signals. Quantization error here compounds along the recurrent state across the sequence rather than staying local, so they're left unquantized despite being Linear layers.
• model.visual.* (the vision tower): untouched, this checkpoint was not calibrated for vision input.

All other Linear layers (self-attention q/k/v/o, linear-attention qkv/z/out projections, and MLP gate/up/down, across all 64 layers) are FP8.

Usage (vLLM)

vllm serve alexliap/Sophea-Genesis-1-FP8-dynamic

Usage (transformers)

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "alexliap/Sophea-Genesis-1-FP8-dynamic", device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("alexliap/Sophea-Genesis-1-FP8-dynamic")

Created with

LLM Compressor v0.13.0, FP8_DYNAMIC scheme.

from llmcompressor.modifiers.quantization import QuantizationModifier

QuantizationModifier(
    targets=["Linear"],
    scheme="FP8_DYNAMIC",
    ignore=[
        "lm_head",
        r"re:.*linear_attn\.in_proj_a$",
        r"re:.*linear_attn\.in_proj_b$",
        r"re:model\.visual\..*",
    ],
)

License

Apache-2.0, inherited from the base model.