A tiny Russian language model (4.7M parameters) based on the Qwen2 architecture. Trained from scratch on a dataset searching for 5MB of toxic website comments.
Specifications
- Parameters: 4.7M
- Layers: 4 | Hidden: 256 | Heads: 4 | FFN: 1024
- Tokenizer: ByteLevel BPE (dictionary: 1000)
- Precision: float32
- Status: pretraining only (no SFT/RLHF)
What is this?
A basic model that statistically continues text. Not a helper – doesn't answer questions or follow instructions. Produces coherent Russian for 1-2 sentences, but semantically drifts further.
Trained on unfiltered internet data, can generate profanity and insults. Not for production.
Why?
Runs on low-end devices (including Android) with ~300 tokens/s. Demonstrates what a very small model can learn.
Limitations
- No instructions
- Small dictionary (1000 tokens)
- Unfiltered data, toxicity
- Not benchmarked (no similar models)
How to install?
It's simple: download the libraries from requirements.txt, run the app.py, and the model is ready for testing.
A little bit about myself
This is a huge step forward in the development of our models. While previous models constantly stumbled, their text fell apart, and their context didn't hold up even with the best of intentions, this is a real breakthrough for us; in fact, this model is the best in its size range.
- Downloads last month
- 73