squeal_ai_20m-instruct

squeal_ai_20m-instruct is a compact ~20M parameter language model, fine-tuned for instruction following on Russian-language data. It belongs to the squeal_ai_ model family by Squeal Studio.

This is the instruct version of squeal_ai_20m-base, fine-tuned with Supervised Fine-Tuning (SFT) on an instruction dataset. No RLHF or DPO alignment stage was applied.

Research and Educational Model. This model is designed for research, educational purposes, and experimentation. Given its parameter count and training data volume, performance on complex instructions or factual tasks will be limited.

Model Description

• Architecture: Qwen2.5-style transformer decoder with GQA (Grouped Query Attention)
• Parameters: ~20M
• Tokenizer: Custom BPE, vocab_size = 24,000
• Context length: 1,024 tokens
• Base model: squeal_ai_20m-base
• Fine-tuning method: Supervised Fine-Tuning (SFT) only, no RLHF/DPO

Architecture Details

Parameter	Value
hidden_size	352
num_hidden_layers	8
num_attention_heads	8
num_key_value_heads	4
intermediate_size	1024
max_position_embeddings	1024
vocab_size	24,000

Training Details

• Base model: squeal_ai_20m-base, pretrained on cultura_ru_edu, fineweb2_ru, and Russian Wikipedia
• SFT dataset: This model was fine-tuned using ru_turbo_alpaca, created by IlyaGusev
  • Dataset License: CC-BY 4.0
  • Modifications: Added examples of self-identification and reformatted to a plain User: / Bot: prompt structure
• Training setup: Trained on a Tesla T4 (fp16) up to step 2,400
• Chat template: none — no chat_template.jinja is included; prompts must be manually formatted as shown below

Usage

This model does not use tokenizer.apply_chat_template. Prompts must follow the plain-text format below:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Squeal-Studio/squeal_ai_20m-instruct")
tokenizer = AutoTokenizer.from_pretrained("Squeal-Studio/squeal_ai_20m-instruct")

prompt = "User: Как поднять настроение?\nBot:"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Scope & Limitations

• Very small parameter count means limited factual knowledge and reasoning ability
• SFT only, no RLHF or DPO, so alignment and instruction-following robustness are limited compared to larger aligned models
• No chat template is provided, the model expects the exact User: ... \nBot: format shown above and may behave unpredictably with other prompt structures
• May produce repetitive, incoherent, or factually incorrect responses
• Not intended for production, safety-critical, or high-stakes use cases
• Best suited for architectural experiments, baseline comparisons, and educational purposes

License

Apache 2.0