V4-Nano-36B-A4B

V4-Nano-36B-A4B is an architecture-only, randomly initialized DeepSeek-V4-style Mixture-of-Experts language-model configuration.

This repository does not contain pretrained weights and is not a usable language model. It publishes a reproducible architecture specification for future training experiments.

Architecture

Field	Value
Total parameters	36,237,680,403
Active parameters per token	4,217,315,091
Transformer layers	29
Hidden size	3,072
Vocabulary size	129,280
Routed experts per layer	80
Shared experts per layer	1
Routed experts selected per token	2
Expert intermediate size	1,536
Query heads	48
KV heads	1
Attention head dimension	512
mHC residual streams	4
Maximum configured context	1,048,576

The layer schedule is:

2 sliding-window attention layers
13 alternating CSA/HCA pairs
1 final sliding-window attention layer

The first three MoE layers use the V4 Hash-MoE architecture. A training implementation must populate their balanced tid2eid routing tables before training. The remaining 26 layers use learned top-2 routing.

Parameter accounting

The count follows the parameter shapes in the Hugging Face DeepseekV4ForCausalLM implementation:

Component	Total parameters	Active per token
Input embedding and untied LM head	794,296,320	794,296,320
Attention, CSA/HCA compressors, and indexers	2,167,002,352	2,167,002,352
Routers, mHC, and normalization	24,463,907	24,463,907
Routed and shared experts	33,251,917,824	1,231,552,512
Total	36,237,680,403	4,217,315,091

The active count includes all dense parameters, one shared expert per layer, and two routed experts per layer. It excludes the training-only MTP auxiliary module because the current Hugging Face causal-LM implementation does not instantiate it.

Why A4B

The original DeepSeek-V4-Flash activates 6 of 256 routed experts per token. This configuration activates 2 of 80:

6 / 256 = 2.344%
2 / 80  = 2.500%

Top-2 therefore preserves the routing sparsity ratio while keeping the architecture practical for a 36B total-parameter model. Eighty experts also partition evenly across both 8-way and 16-way expert parallelism.

Empty initialization

Creating a normally allocated instance requires enough memory for a 36B model. Use a meta device to inspect the skeleton without allocating weight storage:

python init_empty_model.py

The included count_parameters.py computes the parameter count directly from the architecture without allocating the model:

python count_parameters.py

References

• DeepSeek-AI, DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
• Hugging Face Transformers, DeepSeek-V4 implementation

Status

This is an untrained experimental architecture. No benchmark, safety, quality, or deployment claims are made.