squeal_ai_8m-instruct

squeal_ai_8m-instruct is a compact ~8M parameter language model, fine-tuned for instruction following on Russian-language data. It belongs to the squeal_ai_ model family by Squeal Studio.

This is the instruct version of squeal_ai_8m-base, fine-tuned with Supervised Fine-Tuning (SFT) on an instruction dataset. No RLHF or DPO alignment stage was applied.

Research and Educational Model. This model is designed for research, educational purposes, and experimentation. Given its parameter count and training data volume, performance on complex instructions or factual tasks will be limited.

Model Description

• Architecture: Qwen2.5-style transformer decoder with GQA (Grouped Query Attention)
• Parameters: ~8M
• Tokenizer: Custom BPE, vocab_size = 24,000
• Context length: 1,536 tokens
• Base model: squeal_ai_8m-base
• Fine-tuning method: Supervised Fine-Tuning (SFT) only, no RLHF/DPO

Architecture Details

Parameter	Value
hidden_size	192
num_hidden_layers	8
num_attention_heads	6
num_key_value_heads	3
intermediate_size	512
max_position_embeddings	1536
vocab_size	24,000

Training Details

• Base model: squeal_ai_8m-base, pretrained on Russian Wikipedia + OpenSubtitles (RU)
• SFT dataset: publicly available Russian-language instruction dataset from Hugging Face
• Training setup: Trained on a Tesla T4 (fp16) up to step 1400
• Chat template: included (chat_template.jinja) for use with tokenizer.apply_chat_template

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Squeal-Studio/squeal_ai_8m-instruct")
tokenizer = AutoTokenizer.from_pretrained("Squeal-Studio/squeal_ai_8m-instruct")

prompt = "<|user|>\nКак поднять настроение?\n<|assistant|>\n"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Scope & Limitations

• Very small parameter count means limited factual knowledge and reasoning ability
• SFT only — no RLHF or DPO, so alignment and instruction-following robustness are limited compared to larger aligned models
• May produce repetitive, incoherent, or factually incorrect responses
• Not intended for production, safety-critical, or high-stakes use cases
• Best suited for architectural experiments, baseline comparisons, and educational purposes

License

Apache 2.0