Talos Mini 1M — OASST1
Experimental 1,000,320-parameter Talos Mini language model trained on a small subset of OpenAssistant/oasst1.
This repository contains the native Talos checkpoint and native Talos Byte-Level BPE tokenizer.
Model specifications
- Parameters: 1,000,320
- Architecture: TalosGPT decoder-only transformer
- Hidden size: 128
- Layers: 3
- Attention heads: 8
- KV heads: 4
- Head dimension: 16
- Dense FFN intermediate size: 512
- Vocabulary size: 1,024
- Maximum sequence length: 512
- Tokenizer merges: 764
- Training data: 1,800 OASST1 messages
- Evaluation data: 200 held-out OASST1 messages
- Training sequence length: 64
- Training batch size: 4
- Optimizer: AdamW
- Learning rate: 3e-3
Training
The model was trained using the existing Talos training stack.
Published checkpoint:
step-26925.pt
The checkpoint contains the native Talos model state and training metadata.
Results
At step 26,925:
- Validation loss: approximately 2.254
- Validation perplexity: approximately 9.52
- Validation next-token accuracy: approximately 35%
- Evaluated tokens: 102,816
These results come from a small research experiment and are not a benchmark of general language-model capability.
The model can generate recognizable language patterns, but it is not a coherent general-purpose conversational model.
Tokenizer
The tokenizer is the native Talos ByteLevelBPETokenizer.
It is not a Hugging Face GPT-2 tokenizer conversion.
The tokenizer uses Talos-native byte IDs and merge IDs.
- Vocabulary: 1,024
- Merges: 764
Status
Demonstrated
- 1M-parameter Talos model construction
- Native Talos tokenizer
- OASST1 data loading
- Train/evaluation split
- Training
- Checkpoint saving/loading
- Held-out evaluation
- Text generation
- KV-cache equivalence testing
- Greedy generation
Limitations
This is an experimental dev-scale model trained on a small dataset and short sequence length.
It is not intended to compete with modern general-purpose language models.
The 1M model is a scaling experiment between the 254K Talos Mini prototype and larger future Talos configurations.
Long-term Talos scaling targets are research goals and are not represented here as demonstrated capabilities.
- Downloads last month
- 119