Fallen1:800M
Architecture: original decoder-only Transformer, 817,387,008 parameters. 28 layers, width 1536, 16 heads, SwiGLU FFN 4096, RoPE, RMSNorm, tied embeddings, context 256.
IMPORTANT: Fresh weights trained from scratch on English FineWeb-Edu (language modeling) and Smol-SmolTalk (chat). This model is experimental. A public checkpoint is not a guarantee of fluent answers. Consult training_state.json to check its training phase and token counts.
Tokenizer: new ByteLevel BPE with 16,000 IDs. Chat prompts use <|bos|><|user|>question<|eot|><|assistant|> and responses end in <|eot|>.
Training sources: https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu and https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk. Respect upstream dataset terms. Training uses a single Kaggle GPU and restarts across sessions.
Limitations: no independent human evaluation; the model may hallucinate, repeat, respond off topic or produce unsafe text. Not a reliable source of professional advice.
Inference: see chat_inference.py (download model files from this repository).
- Downloads last month
- 155