Instructions to use akoumpa/Moonlight-V4-16B-A3B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use akoumpa/Moonlight-V4-16B-A3B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="akoumpa/Moonlight-V4-16B-A3B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("akoumpa/Moonlight-V4-16B-A3B")
model = AutoModelForCausalLM.from_pretrained("akoumpa/Moonlight-V4-16B-A3B", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use akoumpa/Moonlight-V4-16B-A3B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "akoumpa/Moonlight-V4-16B-A3B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "akoumpa/Moonlight-V4-16B-A3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/akoumpa/Moonlight-V4-16B-A3B

SGLang

How to use akoumpa/Moonlight-V4-16B-A3B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "akoumpa/Moonlight-V4-16B-A3B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "akoumpa/Moonlight-V4-16B-A3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "akoumpa/Moonlight-V4-16B-A3B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "akoumpa/Moonlight-V4-16B-A3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use akoumpa/Moonlight-V4-16B-A3B with Docker Model Runner:
```
docker model run hf.co/akoumpa/Moonlight-V4-16B-A3B
```

Moonlight-V4-16B-A3B

A proposed, untrained DeepSeek-V4-architecture configuration at Moonlight-16B-A3B scale (no weights). Moonlight-16B-A3B is the DeepSeek-V3 architecture at "DeepSeek-V3-Small" scale (2048 hidden, 27 layers, 64 experts), trained by Moonshot with Muon. This config is the same idea for DeepSeek-V4: it keeps Moonlight's width, depth, expert count and expert size, and swaps in every V4 architectural change: hybrid Compressed Sparse / Heavily Compressed Attention, manifold-constrained hyper-connections (mHC), a hash-routed first MoE layer, shared-KV MQA with a grouped low-rank output projection, sqrt-softplus routing, attention sinks and clamped SwiGLU. COMPARISON.md in this repo is the full side-by-side analysis against Moonlight; this card is the summary.

	total	non-embedding	activated / token	activated non-embedding
Moonlight-V4-16B-A3B	16,527,519,335 (16.528B)	15.856B	2.980B	2.309B
Moonlight-16B-A3B (for reference)	15.960B	15.289B	2.915B	2.244B

Counts are computed from the tensor shapes of the released implementations (count_params.py, which also reproduces the published sizes of Moonlight, DeepSeek-V3, V4-Flash and V4-Pro) and were confirmed by instantiating DeepSeek's reference inference/model.py and transformers' DeepseekV4ForCausalLM on the meta device.

Lineage

model	architecture	size	status
Moonlight-16B-A3B (Moonshot)	DeepSeek-V3	15.96B / 2.92B active	released, trained with Muon on 5.7T tokens
Moonlight-V4-16B-A3B	DeepSeek-V4 at Moonlight's width / depth / experts	16.528B / 2.980B active	this repo (config only)
`akoumpa/Moonlight-V4-1B-h16d256`	same, shrunk to 1024 hidden / 15 layers / 32 experts, 16 x 256 heads	1.00B / 0.54B active	config + trainable recipe
`akoumpa/Moonlight-V4-1B-h16d256-r8`	as above with ratio-8 compression and no indexer	0.98B / 0.52B active	config + trainable recipe

Architecture

component	setting	vs Moonlight-16B-A3B
hidden / layers / vocabulary	2048 / 27 / 163840	unchanged
attention schedule (`compress_ratios`)	`[0, 0, (4, 128) x 12, 4]`: 2 sliding-window, 13 CSA (ratio 4, overlapped, with indexer), 12 HCA (ratio 128)	full causal MLA on every layer
attention	shared-KV MQA: 16 query heads x `head_dim` 512 (last 64 dims RoPE), one 512-dim entry is both key and value	MLA, 16 heads of 192 (q/k) and 128 (v), 576-dim latent per token
query path	`q_lora_rank` 512 (hidden/4, Flash's ratio); shared with the indexer	direct `q_proj`
output projection	grouped low-rank: `o_groups` 2 (8 heads per group, as in Flash and Pro) x `o_lora_rank` 1024; the per-group `[1024, 4096]` matrix is identical to V4-Flash/Pro	dense `o_proj`
sliding window / attention sinks	128 tokens on every layer / one learnable sink logit per head	none
lightning indexer	64 heads x 128 dims, `index_topk` 512 (V4-Flash values)	none
MoE (every layer)	64 routed experts x 1408, top-6, 1 shared expert; `sqrtsoftplus` scoring, aux-loss-free bias, `routed_scaling_factor` 2.441	64 x 1408 top-6, 2 shared experts, sigmoid scoring, factor 2.446
early layers	first 1 MoE layer hash-routed by token id (`tid2eid`)	1 dense layer (`first_k_dense_replace` 1)
residual stream	mHC, `hc_mult` 4, 20 Sinkhorn iterations	plain residual
SwiGLU clamp	`swiglu_limit` 10.0	none
positions	1,048,576 tokens: theta 10000 on sliding layers, 160000 on compressed layers, YaRN x16 from 65,536 (V4 values)	8,192 tokens, theta 50,000
MTP	none (Moonlight dropped it); V4's depth-1 MTP would add 0.590B total / 88M active	none
`rms_norm_eps`	1e-06	1e-5

Per-layer parameters: attention 18.9M (sliding) / 28.4M (CSA: core 18.9M + compressor 4.2M + indexer 5.4M) / 21.0M (HCA); MoE 562.4M total, 60.7M activated (each expert 8.7M); mHC 393,270. Layer totals: sliding 581.7M / CSA 591.3M / HCA 583.9M, activated 80.0M / 89.5M / 82.1M. Embedding and head are 335.5M each.

Where the budget moved relative to Moonlight: all 27 layers are MoE (Moonlight's layer 0 is a 69M dense MLP), the per-token FFN shrinks from 8 to 7 expert-widths (one shared expert instead of two), and attention roughly doubles (0.66B vs 0.37B) because a 512-wide shared-KV head with the low-rank q/o path, compressors, indexers and sinks is heavier than 16 MLA heads. Net: activated parameters within 3% of Moonlight's, hence the same 16B-A3B name.

Long-context efficiency

KV cache per sequence (Moonlight: bf16 MLA latent; this config: FP8 non-RoPE dims + bf16 RoPE dims + FP4 indexer keys):

context	Moonlight-16B-A3B	Moonlight-V4-16B-A3B
8K	243.0 MiB	18.6 MiB
64K	1.90 GiB	135.3 MiB
1M	30.38 GiB	2.09 GiB

Core-attention FLOPs per generated token: 2.26 GF vs 0.79 GF at 8K, 290 GF vs 59.4 GF at 1M (linear layers: 5.16 vs 5.29 GF). A CSA query attends to 128 window entries + 512 selected compressed entries, an HCA query to 128 + L/128, so attention cost is nearly flat in context length.

Files

file	purpose
`config.json`	Hugging Face config (same key set as `deepseek-ai/DeepSeek-V4-Flash`); loads natively in transformers >= 5.8
`inference_config.json`	the same model in the key format of DeepSeek's reference `inference/model.py` (`n_mtp_layers` 0)
`COMPARISON.md`	full comparison with Moonlight-16B-A3B: per-layer tensor inventory, parameter breakdown, KV / FLOP tables, design decisions
`count_params.py`, `reference/*.json`	parameter / KV-cache / FLOP calculator plus the upstream configs it validates against (`python3 count_params.py`)
`tiktoken.model`, `tokenization_moonshot.py`, `tokenizer_config.json`	Moonshot tokenizer (MIT), copied from `moonshotai/Moonlight-16B-A3B`
`training/init_utils.py`	from-scratch initialisers for the hash-routing table and the mHC mixers (see below)

Loading

import torch
from transformers import DeepseekV4Config, DeepseekV4ForCausalLM
cfg = DeepseekV4Config.from_pretrained("akoumpa/Moonlight-V4-16B-A3B")        # transformers >= 5.8; legacy keys are folded into layer_types / rope_parameters
with torch.device("meta"):
    model = DeepseekV4ForCausalLM(cfg)                    # 16.528B parameters

NeMo Automodel: nemo_automodel.components.models.deepseek_v4.config.DeepseekV4Config.from_pretrained("akoumpa/Moonlight-V4-16B-A3B") with NeMoAutoModelForCausalLM.from_config (the native implementation supports FSDP2 / EP / PP / CP and TileLang kernels). DeepSeek reference code: inference_config.json is a drop-in ModelArgs file.

Tokenizer

The config uses Moonlight's vocabulary: 163,840 embedding rows, of which 163,593 ids are defined by the tokenizer (163,584 BPE tokens + 9 special tokens; BOS 163584, EOS 163585; the remaining rows are padding, as in Moonlight). Moonshot's tokenizer files (tiktoken.model, tokenization_moonshot.py, tokenizer_config.json, MIT licence) are bundled. transformers >= 5.8 builds a fast tokenizer directly from tiktoken.model and reproduces moonshotai/Moonlight-16B-A3B's encodings exactly (verified on English, Chinese, code and special-token probes); older transformers versions use the bundled remote-code class and need trust_remote_code=True and tiktoken:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("akoumpa/Moonlight-V4-16B-A3B")

Training notes

No weights. Everything here is architecture-derived. Training this size needs a cluster; the 1B repos above ship a recipe that runs on two 48 GB GPUs and exercises the same code paths.
From-scratch initialisation. Released implementations only ever load checkpoints: the hash-routing table tid2eid is created as zeros (every token to expert 0) and NeMo Automodel leaves the mHC mixer tensors uninitialised. training/init_utils.py fills a balanced token-id hash and applies transformers' mixer init.
Indexer. The lightning indexer's top-k has no gradient path; DeepSeek trains dense for the first 1T tokens (4K context) and distils the indexer when sparse attention is switched on at 64K. With index_topk 512 the selection becomes active beyond 2K tokens of context.
Positions. The config carries V4's 1M-token YaRN setup. For a Moonlight-style 8K run, set max_position_embeddings 8192 and drop rope_scaling; V4 itself extends 4K -> 16K -> 64K -> 1M.
Optimizer. Both Moonlight and DeepSeek-V4 train with Muon (AdamW for embeddings, head, norms and mHC static parameters); V4 uses hybrid Newton-Schulz and no QK-clip.
transformers' V4 implementation is inference-oriented (no causal mask on appended compressed entries without a cache, per-query gathered keys); use NeMo Automodel or the reference code for training.

Design decisions (and the alternatives)

16 heads, not 32: 32 heads would push activated parameters 23% above Moonlight's.
o_lora_rank 1024 keeps V4's exact per-group projection shape; 512 would land exactly on Moonlight's 2.24B activated non-embedding parameters at the cost of a different projection.
Indexer 64 x 128 and index_topk 512 are V4-Flash constants (Pro uses top-k 1024).
One hash layer mirrors Moonlight's single dense layer; V4-Flash/Pro use three.
MTP off (Moonlight precedent); enabling it requires appending a 0 to compress_ratios (V4 hub configs carry num_hidden_layers + num_nextn_predict_layers entries, the MTP block being a sliding-window layer).
routed_scaling_factor 2.441 comes from Moonlight's RMS-matching recipe applied to sqrt-softplus with 64 experts / top-6; V4 shipped tuned values (1.5 Flash, 2.5 Pro).

Limitations and notes

Untrained configuration; numbers are analytic or from meta-device instantiation.
Not affiliated with Moonshot AI or DeepSeek; the name only marks the lineage.

References

Liu et al., Muon is Scalable for LLM Training (Moonlight), arXiv:2502.16982
DeepSeek-AI, DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence, arXiv:2606.19348
DeepSeek-AI, DeepSeek-V3 Technical Report, arXiv:2412.19437
Xie et al., Manifold-Constrained Hyper-Connections (mHC), 2026; Roller et al., Hash Layers for Large Sparse Models, NeurIPS 2021

Downloads last month: 261

Papers for akoumpa/Moonlight-V4-16B-A3B