AT-Neo-100M

A 100.7M-parameter decoder-only language model, trained from scratch. It is a base model (no instruction tuning): it continues text rather than following instructions.

Architecture	Llama-style: RMSNorm, RoPE, SwiGLU MLP, grouped-query attention, tied embeddings
Layers / width	12 layers, d_model 768, MLP 2048
Heads	12 query heads, 4 key/value heads
Context length	1024 tokens
Vocabulary	32,768-token byte-level BPE, trained on the same data
Training	9,537 steps, 5.00B tokens, best validation loss 3.036
HellaSwag (0-shot, 10,042 items)	acc_norm 29.4% · acc 28.4% (random 25%)

Training data

FineWeb-Edu (sample-10BT), licensed ODC-By 1.0 by Hugging Face.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("ashokku/at-neo-100m")
model = AutoModelForCausalLM.from_pretrained("ashokku/at-neo-100m")
ids = tok("The theory of evolution", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=60, do_sample=True, temperature=0.8, top_k=50)
print(tok.decode(out[0]))

Limitations

Small base model: it produces fluent text but often makes up facts, and it has not been safety-tuned. Not suitable for production use without further fine-tuning and evaluation.