llm8-chat-417m

An experimental 417M-parameter decoder-only transformer with a short chat fine-tune on top. It exists to test an architecture, not to be useful: the training budget was about 2.3 billion tokens of pretraining and 0.2 billion of chat fine-tuning, far too little for a model of this size, and it shows. Expect confident nonsense, repetition, and wrong code. Nothing it says should be relied on.

This repository is published for research and educational use only, under a non-commercial license, with no warranty of any kind.

Try it

pip install "transformers>=5" torch safetensors
python chat.py                      # interactive, sampling defaults
python chat.py --greedy --max-new-tokens 256
python chat.py --prompt "Give me three tips for sleeping better."

chat.py downloads the weights on first use, picks cuda, mps or cpu, keeps the conversation within the 2,048-token context by dropping the oldest exchanges, and streams the reply. /reset clears the conversation, /quit exits.

In code:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "ZdsAlpha/llm8-chat-417m"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.float32).eval()

messages = [{"role": "user", "content": "Why is the sky blue?"}]
input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
output = model.generate(input_ids, max_new_tokens=256, do_sample=True, temperature=0.7, top_p=0.9, top_k=50)
print(tokenizer.decode(output[0, input_ids.shape[1]:], skip_special_tokens=True))

The model is a custom architecture, so loading needs trust_remote_code=True; the code it runs is the two short files in this repository. Weights are stored in bfloat16; use dtype=torch.bfloat16 on a GPU and torch.float32 elsewhere.

Chat format

The tokenizer's chat template renders a conversation as

<｜begin▁of▁sentence｜><｜User｜>{user turn}<｜Assistant｜>{assistant turn}<｜end▁of▁sentence｜>...

with <｜Assistant｜> appended as the generation prompt. Only user and assistant roles are supported; there is no system prompt. The model ends its turn with <｜end▁of▁sentence｜>, which is also the EOS and padding token.

Greedy decoding tends to collapse into repetition within a few sentences; the sampling defaults (temperature 0.7, top-p 0.9, top-k 50) are the usable mode and are what generation_config.json carries.

Training loss

Pretraining on the left, chat fine-tune on the right. The faint line is the raw per-step train loss, the solid line its moving average, and the dots the held-out validation loss, scored on the same fixed set every time.

run	tokens	validation loss, first to last	final top-1 / top-5
pretraining, 30 epochs	2.33B	3.883 to 2.882	45.1% / 66.8%
chat fine-tune, 3 epochs	0.18B	2.570 to 2.038	55.0% / 78.6%

Validation is still falling at the end of both runs, with no sign of overfitting: the model is undertrained, not saturated. The step in the pretraining train curve near 1.9B tokens is where the training context was raised from 512 to 2,048 tokens and the learning rate lowered; validation is measured the same way before and after, so its curve shows no step.

Architecture

parameters	416,630,784 unique (132M of them in the tied embedding, counted once)
layers	24
width	1,024
attention	8 heads, per-head RMSNorm on Q, K and V, RoPE (θ = 100,000), no output projection
MLP	SwiGLU, hidden 2,816
residual	zero-initialised per-channel LayerScale on both branches; the attention branch takes the raw residual, the MLP branch is RMSNormed
head	RMSNorm, a linear readout, then the tied embedding
context	2,048 tokens
vocabulary	129,280 byte-level BPE tokens

The tokenizer is the byte-level BPE from deepseek-ai/DeepSeek-V4.1-Flash (MIT licensed), with this repository's own chat template.

Limitations

• Extremely undertrained for its size. Factual claims are frequently wrong and code is unreliable.
• English only, no instruction following beyond the shape of an answer, no safety tuning.
• Not suitable for any production, advisory, or decision-making use.

License

The weights and accompanying files are released under CC BY-NC 4.0: attribution required, no commercial use. Provided as-is, with no warranty, for research and educational purposes.