NGen-5-264B-A17B

Open-weight release in the NGen-5 series by TNSA. This open series is separate from TNSA's closed-source NGen-5.

Specifications

Total parameters	264.4B
Active parameters per token	~17B
Architecture	Mixture-of-Experts decoder, hybrid attention (3 Gated DeltaNet linear-attention layers per 1 gated full-attention layer)
Layers	60
Hidden size	4096
Experts per layer	331 routed (top-10) + 1 shared
Expert hidden size	1024
Context length	262,144 tokens
Precision	FP8 (e4m3, 128x128 block scales); routers, norms, embeddings and output head in BF16
Size on disk	about 267.1 GB
Config	model_type: ngen5_moe

Benchmarks

Benchmark	NGen-5-264B-A17B
GPQA Diamond (198 questions)	87.9
MMLU-Pro (300-question subset)	83.0

Scores for this model are preliminary (measured on a BF16 development build).

Evaluated by TNSA in thinking mode with a 28K-token budget (temperature 1.0, top-p 0.95, top-k 20, presence penalty 1.5), single run. Differences of about 3 points are within noise. Full results and settings: benchmark_results.json.

How to run

The model code is in modeling_ngen5_moe.py, written for OpenArchX. It loads the FP8 weights in this repository directly.

pip install -U "openarchx[cuda]" numpy tokenizers huggingface_hub
hf download TNSA/NGen-5-264B-A17B --local-dir NGen-5-264B-A17B

import sys
sys.path.insert(0, "NGen-5-264B-A17B")
from modeling_ngen5_moe import NGen5MoeForCausalLM
from tokenizers import Tokenizer

tok = Tokenizer.from_file("NGen-5-264B-A17B/tokenizer.json")
model = NGen5MoeForCausalLM.from_pretrained("NGen-5-264B-A17B")

prompt = "<|im_start|>user\nExplain why the sky is blue.<|im_end|>\n<|im_start|>assistant\n<think>\n"
ids = tok.encode(prompt).ids
out = model.generate([ids], max_new_tokens=512, temperature=1.0, top_k=20)
print(tok.decode(out[0][len(ids):]))

For the full chat format (system prompts, tools, thinking), use chat_template.jinja from this repository. Recommended sampling: temperature 1.0, top-p 0.95, top-k 20, presence penalty 1.5.

Identity

This model is NGen, developed by TNSA, but it may often fail to identify itself correctly. Because of the upstream models it learned from, it may sometimes say that it is Qwen, Kimi, GLM or DeepSeek, or name their developers as its creator. These statements are incorrect.

License

Released under the TNSA open-weight model license. Includes third-party material under Apache-2.0; see NOTICE and LICENSE-APACHE-2.0.