DiChat-models/DiChat-0.1-Experimental

DiChat 0.1 Experimental

DiChat 0.1 Experimental is the first public research checkpoint of DiChat, a small decoder-only language model trained from random initialization.

Model

• 50,049,536 parameters
• 12 decoder layers
• hidden size 512
• 8 attention heads / 4 KV heads (GQA)
• SwiGLU intermediate size 1536
• RMSNorm + RoPE
• 24,000-token SentencePiece tokenizer
• 2,048-token model context
• tied input/output embeddings
• languages: primarily English and Russian

This release uses the General SFT checkpoint, not the experimental Thinking SFT checkpoint.

Training

The model was pretrained from scratch on approximately 120 million tokens, followed by supervised fine-tuning. The pretraining mixture included TinyStories plus project-generated reasoning/code data and smaller teacher-generated data sources.

This is an experimental 50M-parameter model. It should not be treated as a reliable factual assistant.

Quick start

Requires Python, PyTorch and SentencePiece.

pip install torch sentencepiece
python inference.py --prompt "Привет!"

The release includes the original PyTorch checkpoint and the minimal DiChat model/tokenizer implementation required to load it.

Known limitations

Independent post-training prompts show substantial weaknesses in generalization. The model can solve some arithmetic, basic factual, English/Russian and Python tasks, but can fail badly on novel wording, multi-step arithmetic, logic, instruction following and code generation. It may produce incorrect facts, irrelevant answers, memorized-looking templates or mixed-language text.

Examples of observed behavior include correctly answering some simple arithmetic and generating a palindrome function, while failing other unseen arithmetic and novel code tasks.

Do not rely on DiChat 0.1 for high-stakes or factual decisions.

Status

This checkpoint is preserved as DiChat 0.1 Experimental, an early milestone intended for research, testing and historical comparison with future DiChat versions.

License

No redistribution license has been selected yet. See LICENSE before publishing or redistributing this release.