YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
ASTRA 400M — 20K Checkpoint
Private research release of the ASTRA 400M language-model checkpoint.
Model
- Parameters: 400,363,776
- Vocabulary: 32,768
- Maximum configured context: 8,192
- d_model: 1,152
- Attention heads: 32
- Layers: 21
- Feed-forward dimension: 3,456
- Dropout: 0
- RoPE base: 10,000
- Checkpoint: 20,000 training steps
Files
model/
ASTRA_400M_CONTINUATION_020000.pt
model_v74.py
tokenizer/
tokenizer.json
config/
final_config.py
evaluation/
validation_results.json
SHA256SUMS.json
README.md
Validation
The 20K checkpoint was evaluated on the same held-out validation-token cache used for the random baseline.
| Model | Loss | Perplexity |
|---|---|---|
| Fresh random model | 10.624667 | 41137.1349 |
| ASTRA 20K | 4.418812 | 82.9976 |
Loss reduction: 58.41%.
Perplexity reduction: 99.80%.
Validation trajectory
| Step | Loss | PPL |
|---|---|---|
| 1,000 | 7.066085 | 1171.5519 |
| 6,216 | 5.014627 | 150.5999 |
| 10,000 | 4.733751 | 113.7214 |
| 15,000 | 4.535191 | 93.2413 |
| 20,000 | 4.418812 | 82.9976 |
Important limitation
Validation results demonstrate substantial learned next-token structure.
They do NOT establish reasoning ability, factual reliability, instruction following, or AGI-level capability.
Qualitative generation tests showed substantial repetition and weak semantic reliability.
This checkpoint should therefore be treated as a research/training artifact rather than a production-ready language model.
Privacy
This repository is intended to remain private.
Do not redistribute the checkpoint without authorization.