squeal_ai_20m-base

squeal_ai_20m-base is a compact ~20M parameter language model pretrained from scratch on Russian-language text. It belongs to the squeal_ai_ model family by Squeal Studio.

This is a base model (pretraining only, without instruction tuning).

Research and Educational Model. This model is designed for research, educational purposes, and experimentation. Given its parameter count and training data volume, performance on complex text generation or factual tasks will be limited.

Model Description

• Architecture: Qwen2.5-style transformer decoder with GQA (Grouped Query Attention)
• Parameters: ~20M
• Tokenizer: Custom BPE, vocab_size = 24,000
• Context length: 1,024 tokens

Architecture Details

Parameter	Value
hidden_size	352
num_hidden_layers	8
num_attention_heads	8
num_key_value_heads	4
intermediate_size	1024
max_position_embeddings	1024
vocab_size	24,000

Training Details

• Dataset: cultura_ru_edu (Russian fineweb-edu analog, 65.3%), fineweb2_ru (20.6%), and Russian Wikipedia (14.2%) — 1,509,932 documents, ~4.7gb, ~2.69B characters, estimated ~700M–1B tokens.
• Preprocessing: Unicode normalization (NFKC), Cyrillic-ratio filtering, degenerate-repetition filtering, digit/punctuation-ratio filtering, Wikipedia paragraph-level chunking, short-utterance grouping, exact deduplication (SHA-256), and approximate deduplication (MinHash/LSH)
• Training Setup: Trained on a Tesla T4 (fp16) up to step 3,200

Evaluation

Step	Epoch	Eval Loss	Perplexity
3200	0.7485	4.057	57.85

Eval loss had largely plateaued by this point in training.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Squeal-Studio/squeal_ai_20m-base")
tokenizer = AutoTokenizer.from_pretrained("Squeal-Studio/squeal_ai_20m-base")

inputs = tokenizer("Привет, как дела?", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Scope & Limitations

• Designed primarily for architectural testing, baseline experiments, and lightweight tasks
• Lacks instruction tuning and high parameter capacity required for multi-turn conversational AI or high-precision factual recall
• Not intended for production environments

License

Apache 2.0