squeal_ai_8m-base

squeal_ai_8m-base is a compact ~8M parameter language model pretrained from scratch on Russian-language text. It belongs to the squeal_ai_ model family by Squeal Studio.

This is a base model (pretraining only, without instruction tuning).

Research and Educational Model. This model is designed for research, educational purposes, and experimentation. Given its parameter count and training data volume, performance on complex text generation or factual tasks will be limited.

Model Description

• Architecture: Qwen2.5-style transformer decoder with GQA (Grouped Query Attention)
• Parameters: ~8M
• Tokenizer: Custom BPE, vocab_size = 24,000
• Context length: 1,536 tokens

Architecture Details

Parameter	Value
hidden_size	192
num_hidden_layers	8
num_attention_heads	6
num_key_value_heads	3
intermediate_size	512
max_position_embeddings	1536
vocab_size	24,000

Training Details

• Dataset: Selected Russian Wikipedia articles and OpenSubtitles (RU) dataset (~350MB raw text, estimated ~50–70M tokens)
• Preprocessing: NFKC normalization, character-ratio filtering, paragraph/document grouping, and MinHash/LSH deduplication
• Training Setup: Trained on a Tesla T4 (fp16) up to step 996

Evaluation

Training and validation loss across pretraining epochs:

Epoch	Eval Loss	Perplexity
1	6.567	711.08
2	5.999	403.03
3 (final)	5.688	295.20

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Squeal-Studio/squeal_ai_8m-base")
tokenizer = AutoTokenizer.from_pretrained("Squeal-Studio/squeal_ai_8m-base")

inputs = tokenizer("Привет, как дела?", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Scope & Limitations

• Designed primarily for architectural testing, baseline experiments, and lightweight tasks
• Lacks instruction tuning and high parameter capacity required for multi-turn conversational AI or high-precision factual recall
• Not intended for production environments

License

Apache 2.0