aday777/qwen4_exp_tiny_fixture

Qwen3.8-Flash-Next tiny architecture fixture (qwen4_exp)

A ~0.29M-parameter random-init TEXT checkpoint plus a reduced config that keeps the real Qwen3.8-Flash-Next qwen4_exp field names, so loaders, quant planners, and CI jobs can exercise the new architecture's config parsing, expert-table sizing, and safetensors load path without the real weights.

What this is

• Base model: Qwen/Qwen3.8-Flash-Next (released 2026-08-24), model_type: qwen4_exp, architectures: ["Qwen4ExpForConditionalGeneration"]. The base is a large multimodal MoE with a hybrid linear+full attention stack, ngram vocab, an attention indexer, PLE, hyper-connections, and an MTP head, so it cannot be instantiated in a unit test, in CI, or on a laptop.
• What this fixture is: a byte-reproducible random-init text checkpoint plus a reduced config that preserves the real top-level wrapper fields (model_type, architectures, image_token_id, video_token_id, vision_start/end_token_id, language_model_only, text_config, vision_config) and the real qwen4_exp_text field names inside text_config — including layer_types (the linear/full attention schedule), linear_*, ngram_*, indexer_*, ple_*, hc_*, the nested mtp block, and rope_parameters.
• How it is changed from the base: same schema, tiny geometry, float32, and a reduced standard-attention + MoE tensor set. It is a schema/load fixture, not a quantization or a distilled model.
• What it is not: not trained, not distilled, not a quality or benchmark claim.
• Why it is useful: it lets you test config parsing, weight-name mapping, expert-table sizing, router/top-k bookkeeping, and safetensors load paths in milliseconds, and it exercises the qwen4_exp multimodal wrapper shape that a plain text-only fixture does not.

Fixture geometry (what this checkpoint actually contains)

Field	Value
num_hidden_layers	4
hidden_size	64
num_attention_heads / num_key_value_heads	4 / 2
head_dim	16
layer_types	linear, linear, linear, full
num_experts	8
num_experts_per_tok	2
shared_expert_intermediate_size	32
moe_intermediate_size	32
dtype	float32
vocab_size	256

Total: 138 tensors, 1,157,376 data bytes = 289,344 float32 parameters.

Intentional omissions (documented, not silent)

• Text-only. No vision encoder or projector tensors; vision_config is a placeholder. A full multimodal loader must supply vision/projector tensors.
• The tensor set is a reduced standard-attention + MoE convention. The real model's linear-attention (conv/ssm) tensors, ngram-embedding parts, attention indexer, PLE, hyper-connection, and MTP tensors are NOT included.
• No lm_head tensor; a loader must tie to model.embed_tokens.weight or supply its own head.
• Tokenizer metadata files are placeholders (no vocab file); use your own tokenizer.

Verification actually performed (stdlib only, no torch in this environment)

• safetensors header parses: 138 tensors, contiguous data_offsets, header padded to 8-byte alignment; __metadata__ records the generator string.
• config.json parses; model_type is qwen4_exp with a nested qwen4_exp_text text_config carrying the real field names listed above.
• checksums.txt records the SHA-256 of every tensor blob.
• Deterministic regeneration: SplitMix64 seed 20260903, Box-Muller normals, scale 0.02, float32 row-major, consumed in sorted-name order.
• Generator script is included as build_fixture.py so the folder can be rebuilt and diffed.

Not yet verified: loading under a specific transformers version (no torch/transformers in the build environment), and whether Qwen4ExpForConditionalGeneration accepts this reduced text-only geometry without the linear-attention / ngram / indexer / PLE / MTP tensors. Treat those as open until run against a real install.

How to run it

Read the tensors with the standard library (no torch needed, matching how this was built):

import json, struct
with open("model.safetensors", "rb") as f:
    n = struct.unpack("<Q", f.read(8))[0]
    header = json.loads(f.read(n))
    # header[name] = {"dtype", "shape", "data_offsets"}; data starts at byte 8+n

Or with the safetensors package:

from safetensors.torch import load_file
tensors = load_file("model.safetensors")   # {name: tensor}

To exercise a real loader, build a config from config.json (the qwen4_exp model type; use AutoConfig.from_pretrained(..., trust_remote_code=True) where needed) and feed these weights in. There is no lm_head tensor, no vision stack, and the tokenizer files are placeholders, so supply your own head/tokenizer/vision.

License

The generated fixture content (random weights, config, scripts) is released under MIT (see LICENSE). The qwen4_exp architecture and config schema belong to the base model Qwen/Qwen3.8-Flash-Next under its own terms, which were not independently re-verified this cycle — check the base repository before redistribution.

Citation

Qwen Team, Qwen3.8-Flash-Next, 2026.

---

If this fixture saved you an afternoon of debugging, you can buy me a beer: BTC bc1q5ayht3fxhj0v95fk0z8l2f6900g3awdsw5842p