aday777/glm_moe_dsa_tiny_fixture

GLM-5.3 tiny architecture fixture (glm_moe_dsa)

A ~0.27M-parameter random-init checkpoint that reproduces the reduced GLM-5.3 MoE config schema so loaders, quant planners, and CI jobs can exercise the new glm_moe_dsa architecture without touching the 753B weights.

Support this work — if this saved you time, donate BTC: bc1q5ayht3fxhj0v95fk0z8l2f6900g3awdsw5842p

What this is

• Base architecture: zai-org/GLM-5.3 (released 2026-08-25), model_type: glm_moe_dsa, architectures: ["GlmMoeDsaForCausalLM"], MIT license.
• What it is: a byte-reproducible random-init checkpoint plus a reduced config that keeps the real field names and the dense-to-MoE layer schedule (first_k_dense_replace, moe_layer_freq, routed experts + shared experts).
• What it is not: not trained, not distilled, not a quality or benchmark claim, and not a quantization of anything.
• Why it is useful: the base is 753,329,940,480 parameters (Hub safetensors metadata), so it cannot be instantiated in a unit test, in CI, or on a laptop. This fixture lets you test config parsing, weight-name mapping, expert-table sizing, router/top-k bookkeeping, and safetensors load paths in milliseconds.

Geometry (base vs fixture)

Field	Base GLM-5.3	This fixture
num_hidden_layers	78	4
hidden_size	6144 (read partially from base config; verify)	64
num_attention_heads / num_key_value_heads	64 / 64	4 / 4
head_dim	192	16
n_routed_experts	256	8
num_experts_per_tok	8	2
n_shared_experts	1	1
n_group	1	1
first_k_dense_replace	3	1
moe_intermediate_size	2048	32
intermediate_size (dense)	not captured	128
num_nextn_predict_layers	1	0
dtype	bfloat16	float32
vocab_size	154820-class	256

Intentional omissions (documented, not silent)

• No DeepSeek-style sparse-attention (DSA) indexer tensors.
• No MTP / next-n-predictor head (num_nextn_predict_layers: 0).
• No lm_head tensor; a loader must tie to model.embed_tokens.weight or supply its own head.
• Tokenizer metadata files are placeholders (no vocab file); use your own tokenizer.

Verification actually performed (stdlib only, no torch in this environment)

• safetensors header parses: 113 tensors, 1,097,984 data bytes = 274,496 float32 parameters, contiguous data_offsets, header padded to 8-byte alignment.
• checksums.txt records the SHA-256 of every tensor blob.
• Deterministic regeneration: SplitMix64 seed 20260901, Box-Muller normals, scale 0.02, float32 row-major, consumed in sorted-name order.
• Generator script is included as build_fixture.py so the folder can be rebuilt and diffed.

Not yet verified: loading under a specific transformers version (no torch/transformers in the build environment), and whether GlmMoeDsaForCausalLM accepts this reduced geometry without extra fields. Treat those as open until run against a real install.

How to use

Read the tensors with the standard library (no torch needed, matching how this was built):

import json, struct
with open("model.safetensors", "rb") as f:
    n = struct.unpack("<Q", f.read(8))[0]
    header = json.loads(f.read(n))
    # header[name] = {"dtype", "shape", "data_offsets"}; data starts at byte 8+n

Or with the safetensors package:

from safetensors.torch import load_file
tensors = load_file("model.safetensors")   # {name: tensor}

To exercise a real loader, build a config from config.json (the glm_moe_dsa model type; use AutoConfig.from_pretrained(..., trust_remote_code=True) where needed) and feed these weights in. There is no lm_head tensor and the tokenizer files are placeholders, so supply your own head/tokenizer.

License

MIT, unchanged from the base model per its Hub metadata. See LICENSE.

Citation

Zhipu AI / Z.ai, GLM-5.3, 2026.

Support this work

If this saved you time or money, consider a donation: Bitcoin: bc1q5ayht3fxhj0v95fk0z8l2f6900g3awdsw5842p