Mitchins/GLM-4.6V-Flash-FP8-Block128

GLM-4.6V-Flash · FP8 Block-128 (compressed-tensors)

Quantized release of zai-org/GLM-4.6V-Flash using compressed-tensors, with the entire multimodal vision stack (model.visual.*) preserved in BF16 and verified with end-to-end image understanding tests.

What was quantized

Component	Quantization	Notes
model.language_model.layers.*	FP8 E4M3 (W8A8, block-128)	All linear projections
model.visual.*	BF16 (unchanged)	Vision encoder + merger/projector
model.language_model.embed_tokens	BF16 (unchanged)	Embedding table
lm_head	BF16 (unchanged)	Logit projection

Scheme: FP8_BLOCK — per-block-128 weight quantization + dynamic per-block-128 activation quantization. Data-free (no calibration dataset required).

240 language-model Linear layers quantized; 181 vision tensors kept at BF16.

Disk size

	Size
Base model (BF16)	~18 GB
This repo (FP8)	12.4 GB

Verification

Tested on an NVIDIA RTX 5090 (32 GB) with:

• transformers==4.57.6
• llmcompressor==0.10.0.1
• vllm==0.19.0
• torch==2.10.0+cu128
• CUDA 12.8

Bridge image test (Sydney Harbour Bridge, Wikimedia Commons):

"The image shows the Sydney Harbour Bridge at night. The bridge is illuminated with lights along the bridge. The background shows the city skyline with buildings illuminated at night."

Model correctly identified the bridge and scene. Vision stack functional end-to-end.

Usage

vLLM (recommended)

vllm serve Mitchins/GLM-4.6V-Flash-FP8-Block128 --dtype bfloat16

The server auto-detects quantization: compressed-tensors from the config.

Transformers (with compressed-tensors)

import torch
from PIL import Image
from transformers import (
    Glm4vForConditionalGeneration,
    Glm4vImageProcessor, Glm4vVideoProcessor, Glm4vProcessor,
    AutoTokenizer,
)

MODEL = "Mitchins/GLM-4.6V-Flash-FP8-Block128"

model = Glm4vForConditionalGeneration.from_pretrained(
    MODEL, dtype=torch.bfloat16, device_map="auto"
)

# Patch rope_scaling for transformers: config stores [8,12,12] (vllm-compatible),
# but transformers doubles it internally so we must pre-double to [16,24,24].
_rs = model.config.text_config.rope_scaling
if _rs and _rs.get("mrope_section") == [8, 12, 12]:
    _tf = [x * 2 for x in _rs["mrope_section"]]
    for _mod in model.modules():
        if hasattr(_mod, "rope_scaling") and isinstance(getattr(_mod, "rope_scaling", None), dict):
            _mod.rope_scaling = {**_mod.rope_scaling, "mrope_section": _tf}

tokenizer       = AutoTokenizer.from_pretrained(MODEL)
image_processor = Glm4vImageProcessor.from_pretrained(MODEL)
video_processor = Glm4vVideoProcessor.from_pretrained(MODEL)
processor = Glm4vProcessor(
    image_processor=image_processor,
    tokenizer=tokenizer,
    video_processor=video_processor,
    chat_template=tokenizer.chat_template,
)

image = Image.open("your_image.jpg").convert("RGB")
messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "text", "text": "Describe this image."},
]}]
text   = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=text, images=[image], return_tensors="pt")
inputs = {k: v for k, v in inputs.items() if k != "token_type_ids"}
inputs = {k: v.to(model.device) for k, v in inputs.items()}

with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=256, do_sample=False)

print(processor.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Known caveats and packaging fixes

Several non-obvious issues had to be resolved to produce this checkpoint. Documenting them here to save future packagers time.

1. Vision tower module path

The model's vision stack lives at model.visual.* (encoder blocks, patch embedding, merger/projector). The llmcompressor ignore list must use:

re:model\.visual.*

Not model.vision_tower.* or model.multi_modal_projector.* — those paths do not exist in this model.

2. rope_parameters vs rope_scaling mismatch

The hub config stores M-RoPE settings in rope_parameters (the vllm convention). Transformers 4.57+ reads rope_scaling["mrope_section"] directly. The two have conflicting constraints:

• vllm asserts sum(mrope_section) == rotary_dim // 2 == 32 → needs [8, 12, 12]
• transformers doubles mrope_section internally before splitting cos/sin, so the stored values must equal rotary_dim // 2 == 32 → also [8, 12, 12] stored, but must be doubled in-memory to [16, 24, 24] before transformers inference

This repo's config.json stores text_config.rope_scaling.mrope_section = [8, 12, 12] (vllm-compatible). When using transformers directly, patch in-memory after loading:

model = Glm4vForConditionalGeneration.from_pretrained(MODEL, dtype=torch.bfloat16, device_map="auto")

# Required: double mrope_section for transformers (it reads rope_scaling and doubles internally)
_rs = model.config.text_config.rope_scaling
if _rs and _rs.get("mrope_section") == [8, 12, 12]:
    _tf = [x * 2 for x in _rs["mrope_section"]]  # [16, 24, 24]
    for _mod in model.modules():
        if hasattr(_mod, "rope_scaling") and isinstance(getattr(_mod, "rope_scaling", None), dict):
            _mod.rope_scaling = {**_mod.rope_scaling, "mrope_section": _tf}

rope_parameters is kept intact (vllm reads it directly and ignores rope_scaling).

3. AutoProcessor returns the wrong type

AutoProcessor.from_pretrained silently returns only a PreTrainedTokenizerFast for this model ID because preprocessor_config.json declares "image_processor_type": "Glm46VImageProcessor" but the transformers registry only knows "Glm4vImageProcessor". Build the processor manually as shown in the usage example above, passing chat_template=tokenizer.chat_template.

4. token_type_ids rejected by model

The tokenizer emits token_type_ids but the model's generate() rejects them. Filter before inference:

inputs = {k: v for k, v in inputs.items() if k != "token_type_ids"}

Reproducibility

To reproduce this checkpoint from the base model:

git clone https://huggingface.co/Mitchins/GLM-4.6V-Flash-FP8-Block128
# or use compress.py from the companion repo
python compress.py        # FP8 (default)
python compress.py --fp4  # NVFP4 (see companion repo)

Full source: compress.py and verify.py are included in this repo.

License

Derived from zai-org/GLM-4.6V-Flash, released under MIT. This quantized derivative inherits the same license.