xslm-70m

A 70,546,560 parameter Llama-style decoder, trained from scratch on one RTX 3090 in 180 minutes. The project exists to teach the full training pipeline: the tokenizer, the data pipeline, the modules, and the training configuration.

Limitations

This model does not hold world knowledge. A model of this size learns grammar, style, and document structure. It invents facts. This is the expected result at this scale, and it is not a defect to report.

For a model that answers questions correctly, use retrieval-augmented generation over a document index, or a QLoRA fine-tune of an existing 1B to 8B base model.

Architecture

Field	Value
Non-embedding parameters	49,575,040
Total parameters	70,546,560
hidden_size	640
num_hidden_layers	10
num_attention_heads	10
intermediate_size	1728
vocab_size	32768
max_position_embeddings	1024
Normalization	RMSNorm
Position encoding	RoPE, theta 10000.0
Activation	SwiGLU
Embeddings	Tied input and output

Training

Field	Value
Corpus	HuggingFaceFW/fineweb-edu, sample-10BT, ODC-By license
Tokenizer	Byte-level BPE, 32768 tokens
Optimizer	AdamW, betas 0.9 and 0.95
Peak learning rate	0.001
Schedule	200 warmup steps, then cosine decay to 0.0001
Tokens for each optimizer step	262,144
Precision	BF16 mixed
Hardware	One RTX 3090

Files

File	Use
model.safetensors	The BF16 weights, for transformers
gguf/xslm-70m-BF16.gguf	The unquantized GGUF file, in the training dtype
gguf/xslm-70m-F16.gguf	The unquantized GGUF file, for the tools that read no BF16
gguf/xslm-70m-Q8_0.gguf	The recommended quantized file
gguf/xslm-70m-Q6_K.gguf	Smaller, with a visible quality cost
gguf/xslm-70m-Q4_K_M.gguf	A demonstration only

At 70M parameters there is no redundancy to absorb quantization error. Q4_K_M often produces broken text. Use Q8_0.

Use

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Alienstro/xslm")
model = AutoModelForCausalLM.from_pretrained("Alienstro/xslm")