aday777/deepseek_v4_tiny_fixture

DeepSeek-V4-Flash-Vision-Exp tiny architecture fixture (deepseek_v4)

A ~0.29M-parameter random-init TEXT checkpoint plus a reduced config that keeps the real DeepSeek-V4-Flash-Vision-Exp deepseek_v4 field names, so loaders, quant planners, and CI jobs can exercise the new architecture's config parsing, expert-table sizing, and safetensors load path without the real weights.

What this is

• Base model: deepseek-ai/DeepSeek-V4-Flash-Vision-Exp (created 2026-08-31), model_type: deepseek_v4, architectures: ["DeepseekV4ForCausalLM"], MIT license. The base is a large multimodal MoE (image-text-to-text) with MLA attention, routed + shared experts, a DSA index, hash layers, an MTP head, hyper-connections (hc_*), a dspark_* block, and a full vision encoder, so it cannot be instantiated in a unit test, in CI, or on a laptop.
• What this fixture is: a byte-reproducible random-init text checkpoint plus a reduced config that preserves the real deepseek_v4 field names — MLA (q_lora_rank, o_lora_rank, o_groups, qk_rope_head_dim), MoE (n_routed_experts, n_shared_experts, num_experts_per_tok, topk_method, scoring_func, routed_scaling_factor, expert_dtype), the DSA/hash knobs (num_hash_layers), the MTP knob (num_nextn_predict_layers), swiglu_limit, sliding_window, and the yarn rope_scaling block.
• How it is changed from the base: same schema, tiny geometry, float32, and a reduced standard-attention + MoE tensor set. It is a schema/load fixture, not a quantization or a distilled model.
• What it is not: not trained, not distilled, not a quality or benchmark claim.
• Why it is useful: it lets you test config parsing, weight-name mapping, expert-table sizing, router/top-k bookkeeping, and safetensors load paths in milliseconds, and it exercises the deepseek_v4 text-config shape that a plain Llama-style fixture does not.

Fixture geometry (what this checkpoint actually contains)

Field	Value
num_hidden_layers	4
hidden_size	64
num_attention_heads / num_key_value_heads	4 / 1
head_dim	16
q_lora_rank / o_lora_rank / o_groups	32 / 32 / 4
n_routed_experts / n_shared_experts	8 / 1
num_experts_per_tok	2
moe_intermediate_size	32
num_nextn_predict_layers (MTP)	0 (omitted)
num_hash_layers	0 (omitted)
dtype	float32
vocab_size	256

Total: 150 tensors, 1,162,496 data bytes = 290,624 float32 parameters.

Intentional omissions (documented, not silent)

• Text-only. No vision encoder or projector tensors. A full multimodal loader must supply vision/projector tensors.
• The tensor set is a reduced standard-attention + MoE convention. The real model's MLA compressed tensors, DSA index tensors, hash-layer tensors, MTP head, and hyper-connection / dspark_* tensors are NOT included.
• No lm_head tensor; a loader must tie to model.embed_tokens.weight or supply its own head.
• Tokenizer metadata files are placeholders (no vocab file); use your own tokenizer.

Verification actually performed (stdlib only, no torch in this environment)

• safetensors header parses: 150 tensors, contiguous data_offsets, header padded to 8-byte alignment; __metadata__ records the generator string.
• config.json parses; model_type is deepseek_v4 with the real field names listed above.
• checksums.txt records the SHA-256 of every tensor blob.
• Deterministic regeneration: SplitMix64 seed 20260903, Box-Muller normals, scale 0.02, float32 row-major, consumed in sorted-name order.
• Generator script is included as build_fixture.py so the folder can be rebuilt and diffed.

Not yet verified: loading under a specific transformers version (no torch/transformers in the build environment), and whether DeepseekV4ForCausalLM accepts this reduced text-only geometry without the MLA / DSA / hash / MTP / vision tensors. Treat those as open until run against a real install.

How to run it

Read the tensors with the standard library (no torch needed, matching how this was built):

import json, struct
with open("model.safetensors", "rb") as f:
    n = struct.unpack("<Q", f.read(8))[0]
    header = json.loads(f.read(n))
    # header[name] = {"dtype", "shape", "data_offsets"}; data starts at byte 8+n

Or with the safetensors package:

from safetensors.torch import load_file
tensors = load_file("model.safetensors")   # {name: tensor}

To exercise a real loader, build a config from config.json (the deepseek_v4 model type; use AutoConfig.from_pretrained(..., trust_remote_code=True) where needed) and feed these weights in. There is no lm_head tensor, no vision stack, and the tokenizer files are placeholders, so supply your own head/tokenizer/vision.

License

The generated fixture content (random weights, config, scripts) is released under MIT (see LICENSE). The deepseek_v4 architecture and config schema belong to the base model deepseek-ai/DeepSeek-V4-Flash-Vision-Exp under its own terms, which were not independently re-verified this cycle — check the base repository before redistribution.

Citation

DeepSeek, DeepSeek-V4-Flash-Vision-Exp, 2026.

---

If this fixture saved you an afternoon of debugging, you can buy me a beer: BTC bc1q5ayht3fxhj0v95fk0z8l2f6900g3awdsw5842p