Duplicated from  brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw

0xSero

/

GLM-5.2-TR3-Vision

like 8

Summary

docker-compose.yml passes the GLM-5.2 sparse-index schedule through
--hf-overrides, but the current multimodal config does not retain those
overrides in the nested text config that vLLM actually uses.

This PR:

• keeps the existing flat --hf-overrides interface;
• forwards use_index_cache and index_topk_pattern through
  Glm5vConfig into text_config;
• rejects malformed schedules before launch; and
• extends packaged validation to cover both the target model and the MTP draft.

There are no vLLM source changes and no changes to model weights, the chat
template, tokenizer, or vision tower.

Where the setting is lost

The serving bridge intentionally separates the multimodal wrapper from its two
backbones:

--hf-overrides
       |
       v
Glm5vConfig
  |-- vision_config --> MoonViT + multimodal projector
  |
  `-- text_config ----> GlmMoeDsaForCausalLM (78 layers)
                              |
                              `--> DeepSeekMTPModel draft config

runtime/glm52_vision/model.py constructs the language model with
hf_config=config.text_config. The speculative-config bridge likewise copies
hf_config.text_config for MTP. That is the correct architecture, but it means
text-only launch overrides must cross the outer Glm5vConfig boundary first.

On the unmodified 2bc4c3b snapshot, the real Transformers loading path shows
that they do not:

outer use_index_cache:      None
inner use_index_cache:      None
outer pattern length:       0
inner index_topk_pattern:   None
inner index_topk_freq:      4
intended pattern prefix:    FFFSSSFSSSFS

The exact 78-layer full/shared map is therefore absent when the
GlmMoeDsaForCausalLM layers are built. vLLM can only fall back to the
frequency-derived schedule instead of consuming the map supplied by the launch
recipe.

Failure surface

This is silent at startup. The model loads, vision preprocessing succeeds, and
short completions can look plausible. In our serving workload, the problem
became visible after prompts crossed multiple chunked-prefill passes: generation
became repetitive or incoherent, and structured agentic tool calls frequently
failed to close.

That initially looked like a Hermes/tool-parser problem because malformed tool
calls were the user-visible symptom. The parser was downstream of the actual
failure: the requested sparse-index schedule had never reached the 78-layer text
backbone.

Tested environment:

• 4x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition
  (96 GB, compute capability 12.0);
• TP4 + DCP4;
• B12X_MLA_SPARSE attention;
• EXL3 weights with NVFP4 DSA KV cache;
• MTP enabled; and
• chunked prefill with a 524,288-token configured maximum context.

Changes

• configuration_glm5v.py
  • adds forwarding properties for use_index_cache and
    index_topk_pattern;
  • validates that the pattern contains one F/S entry per text layer.
• docker-compose.yml
  • preserves the existing flat override JSON;
  • fails before launch unless the schedule contains exactly 78 F/S
    characters.
• runtime/glm52_vision/validation.py
  • loads the config through the real AutoConfig.from_pretrained override
    path;
  • asserts that both the outer interface and text_config expose the same
    values;
  • asserts that the MTP draft receives the same schedule.

Verification

The packaged validation returned:

{
  "architecture": "Glm5vForConditionalGeneration",
  "draft_architecture": "DeepSeekMTPModel",
  "index_topk_pattern_length": 78,
  "pixel_values_shape": [4, 3, 14, 14],
  "vision_entries": 329,
  "projector_entries": 6
}