Talos Mini 254K — OASST1

Parameters: 254,272

Talos Mini is a 254,272-parameter experimental decoder-only language model trained on a 2,000-example subset of OpenAssistant (OASST1).

Model specifications

Parameters: 254,272
Layers: 2
Hidden size: 64
Attention heads: 4
KV heads: 2
Attention: GQA
Vocabulary size: 1,024
Maximum sequence length: 512
Tokenizer: Talos-native BPE

Training

Dataset: OpenAssistant / OASST1
Examples: 2,000
Training split: 1,800
Validation split: 200
Sequence length: 64
Hardware: NVIDIA Tesla T4
Final training step: 7,680
Final training loss: 1.7859
Final validation loss: 1.9177

Training progress

Step	Train Loss	Validation Loss
480	3.0289	2.9331
2400	2.2165	2.3276
3840	2.0245	2.1350
7680	1.7859	1.9177

Files

step-7680.pt — trained Talos Mini checkpoint
tokenizer.json — Talos tokenizer
metrics.json — training metrics
config.json — model configuration and parameter metadata

Limitations

This is a very small experimental model. Its free-running generations are often incoherent or repetitive, and the model should not be compared directly with modern production-scale language models.

The checkpoint is primarily a research and scaling baseline for Talos.

Downloads last month: 169