RetNet-100M (first chat model)

The first chat model in this project, from 2026-08-20. It talks, badly. It is here because it is the starting point of a line that ends at RetNet-0.5B, and because the loss curve of a model this size is worth having.

Code and the engineering log: https://github.com/Ayaka157/RetNet

Configuration

Parameters	100,370,064
Layers	12
d_model	768
Heads	12
Vocab	20000
Context	512
Step	16,000
Validation loss	4.3768 (perplexity ≈ 79.6)

Trained on a mixed Chinese/English dialogue corpus. Peak validation loss was measured on the training machine; the exact corpus revision is recorded in the project's docs/12_训练状态盘点.md.

Provenance

This checkpoint was trained with the handwritten implementation (retnet/model.py) and converted to Hugging Face format with scripts/migrate_to_hf.py. The conversion verifies numerical identity against the original: the maximum absolute difference in logits is 0.000e+00 at sequence lengths 1, 8 and 128, and all 172 state-dict keys load strictly with no renaming.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Ayaka157/RetNet-100M"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.float32).eval()

ids = tok("<bos>用户：你好。\n助手：")["input_ids"]
out = model.generate(torch.tensor([ids]), max_new_tokens=64, temperature=0.7)
print(tok.decode(out[0].tolist()[len(ids):]))

mode in config.json is parallel for inference; the training run used chunkwise.

The chat format is <bos> + optional 设定：... persona block + 用户：... + 助手：... + <eos>, implemented in retnet/chat.py. Keep prompts to the 512-token context window this model was trained on.

License

MIT.