DiChat Mini 10M Base
Experimental 9,804,032-parameter Russian-first decoder-only Transformer, trained from scratch, without pretrained weights or SFT. This is a base completion model, not a chat assistant.
Architecture
12 layers, hidden size 256, 4 heads, SwiGLU width 608, 4096-token ByteLevel BPE vocabulary, context 512, RoPE, RMSNorm, tied input/output embeddings. Checkpoint after 10,000 training steps.
Benchmarks (author-run, exploratory)
| Evaluation | DiChat Mini 10M |
|---|---|
| Russian Stories held-out test perplexity | 13.49 |
| Russian Stories next-token accuracy | 48.76% |
| TinyStories V2 GPT-4 validation (English) perplexity | 30.83 |
| TinyStories English next-token accuracy | 17.98% |
| Bilingual Release Benchmark v1 (Russian, 500 items) | 48.2% |
The bilingual benchmark also scored 51.2% for roneneldan/TinyStories-3M in English. These are custom template-generated cloze comparisons, not standardized leaderboard results. The two models use different languages and tokenizers; results do not establish model superiority. Both scores are near chance. Evaluation scripts and results are included.
Russian test: 64 sampled windows of 256 tokens (16,384 tokens), PPL 13.49. English TinyStories: 128 sampled windows of 256 tokens (32,768 tokens), PPL 30.83. Windows may overlap. Perplexities across different tokenizers are not directly comparable.
Usage
Install torch and tokenizers, then run:
python inference.py
The custom PyTorch architecture is implemented in model.py. This checkpoint is not directly compatible with Transformers AutoModel or Ollama.
Training data and licensing
Training sources:
inkoziev/ru_stories, listed as CC BY-NC 4.0 (non-commercial).Imperius/ru-classic, described by the dataset as CC0; copyright status of individual works has not been independently audited.
The model's release license and downstream commercial-use permissions require separate rights review. No unrestricted commercial-use permission is claimed. The weights do not include copies of the training corpus. Please review the source datasets' terms before redistribution or commercial use.
Limitations
The model can generate Russian-like short stories but often loses narrative consistency, changes subjects, and does not reliably follow instructions or solve unfamiliar reasoning tasks. This is a research checkpoint, not a production assistant.