RetNet-0.5B

A retentive network language model written from scratch, trained on a Chinese chat corpus. This is epoch 3, the best of three training legs by validation loss.

Code, benchmark and the full technical report: https://github.com/Ayaka157/RetNet

Configuration

Parameters	504,179,896
Layers	20
d_model	1408
Heads	22
ffn_dim / value_dim	2816 / 2816
Vocab	20000
Context	2048 (chunk size 128)
Storage dtype	float32
Retention	multi-scale, learnable per-head decay gate (use_gla)
Norm	function-preserving growth layer norm (growth_old_dim 768)

Training

One epoch per leg over chat_pack_v13, 552,330,489 token slots (269,561 rows × 2049, of which 16,126,035 are padding). Each leg restarts a full cosine from 1.1e-4 to 1.1e-5 with 500 warmup steps, AdamW 8-bit, weight decay 0.1, seed 1234.

Leg	GPU	Batch × accum	Steps	Wall clock	Val loss
ep1	RTX 3090 24 GB	4 × 1	67,390	11.316 h	2.1035
ep2	RTX PRO 6000 96 GB	16 × 1	16,848	2.790 h	1.9865
ep3	RTX PRO 6000 96 GB	16 × 1	16,848	2.685 h	1.9459

3.28496 tokens per parameter in total. Each leg's first evaluation lands above the previous leg's best (ep2 by 0.0764, ep3 by 0.1310) before recovering and finishing lower.

Evaluation

bench_v3: 1,020 items, 15 categories, one seed.

Category	n	Score	Category	n	Score
Safety	35	0.2857	Cloze	120	0.0250
Multi-turn	90	0.2667	Bilingual	45	0.0222
Roleplay	100	0.2000	Instruction following	80	0.0125
Consistency	70	0.1429	Language	90	0.0111
Robustness	50	0.1000	Coding	25	0.0000
QA	120	0.1000	Math	25	0.0000
Structured	60	0.0000	Continuation	60	0.0000
Long context	50	0.0000

Weighted total 0.1037 (95% CI 0.032–0.130). Think-leakage 59.61%, truncation 72.94%.

The item budget is 16–160 new tokens while the median training reply is about 570 tokens, so 72.94% of items hit the ceiling. Among the 276 items that were not truncated, the leak rate is 0.00 and the score is 0.15217. One seed only — the per-category numbers carry no statistical power.

What it is and is not good at

It holds a conversation in Chinese, keeps a persona, and handles multi-turn context. It does not do factual recall, arithmetic, structured output or code. Expect fluent but unreliable text.

Usage

trust_remote_code=True is required. The tokenizer is a standard PreTrainedTokenizerFast, so AutoTokenizer loads it with no custom code.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Ayaka157/RetNet-0.5B"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.float32).eval()

prompt = "<bos>用户：你好，介绍一下你自己。\n助手："
ids = tok(prompt)["input_ids"]
out = model.generate(torch.tensor([ids]), max_new_tokens=64, temperature=0.7)
print(tok.decode(out[0].tolist()[len(ids):]))

This packaging was verified before upload: it loads, a bare forward returns (1, L, 20000) logits, and generation produces coherent Chinese.

mode in config.json is set to parallel for inference. The training checkpoints stored chunkwise, which requires the sequence length to be a multiple of chunk_size and so fails on any short prompt. generate() hardcodes parallel for prefill and recurrent for decoding, and training overrides this field, so the change affects nothing except bare forwards. The three paths are mathematically equivalent but not bit-for-bit; the largest absolute difference measured at this scale is 3.395.

Known quirks

• 78 of the 82 growth affine tensors are exactly zero, and 67.08% of all weight elements are zero. The channels added during width growth never unlocked: gamma_new = 0 cuts the gradient to the new inputs, and those inputs are zero, so gamma_new stays zero too. Initializing it to 1.0 would have been just as function-preserving at step 0 and would have let them unlock — that was not tested. See the technical report §2.4.
• The msr.gamma values of heads 16–21 sit at 0.5430, which matches AdamW's decoupled weight decay over the integrated learning-rate schedule to within 0.02%. Those heads never received gradient; weight decay moved them.
• Tokenizer, chat_pack_v13 and training logs are not distributed.

License

MIT. Training data comes from public corpora (r1_110k_sft, i_wonder_why, better_ruozhiba, plus an earlier mixed base); check those sources before reuse.