squeal_lstm_5m

squeal_lstm_5m is a compact ~5.8M parameter language model pretrained from scratch on Russian-language text, based on LSTM architecture.

This is a base model (pretraining only, without instruction tuning).

Research and Educational Model. This model is designed for research, educational purposes, and experimentation. Given its parameter count and training data volume, performance on complex text generation or factual tasks will be limited.

Model Description

• Architecture: LSTM decoder with projection layer
• Parameters: ~5.8M
• Tokenizer: Custom Unigram SentencePiece, vocab_size = 4,000, character_coverage = 0.9999
• Context length: 256 tokens

Architecture Details

Parameter	Value
embed_dim	256
hidden_dim	512
num_layers	2
dropout	0.3
vocab_size	4,000

Training Details

• Dataset: 210,000 documents, ~500MB — opensubtitles (23.8%), habr (14.3%), taiga_proza (14.3%), ru_news (14.3%), cultura_ru_edu (11.9%), fineweb2_ru (11.9%), wikipedia (9.5%)
• Preprocessing: Unicode normalization (NFKC), Cyrillic-ratio filtering, degenerate-repetition filtering, digit/punctuation-ratio filtering, Wikipedia, Fineweb2, cultura paragraph-level chunking, short-utterance grouping, exact deduplication (SHA-256), and approximate deduplication (MinHash/LSH)
• Training Setup: Trained on a Tesla T4 (fp16), 2 epochs, up to step 5,870
• Sequence length: 256 tokens
• Batch size: 128
• Learning rate: 3e-3 (cosine schedule)

Evaluation

Step	Epoch	Eval Loss	Perplexity
1000	0.34	4.408	82.2
2000	0.68	3.994	54.1
3000	1.02	3.865	47.8
4000	1.36	3.800	44.7
5000	1.70	3.769	43.4
5870	2.00	3.763	43.1

Usage

import torch
import torch.nn as nn
from transformers import AutoTokenizer, PreTrainedModel, PretrainedConfig
from transformers.modeling_outputs import CausalLMOutput


class LSTMConfig(PretrainedConfig):
    model_type = "lstm_lm"

    def __init__(self, vocab_size=4000, embed_dim=256, hidden_dim=512, num_layers=2, dropout=0.3, **kwargs):
        super().__init__(**kwargs)
        self.vocab_size = vocab_size
        self.embed_dim = embed_dim
        self.hidden_dim = hidden_dim
        self.num_layers = num_layers
        self.dropout = dropout


class LSTMLanguageModel(PreTrainedModel):
    config_class = LSTMConfig
    tied_weights_keys = []

    @property
    def all_tied_weights_keys(self):
        return {}

    def __init__(self, config):
        super().__init__(config)
        self.embedding = nn.Embedding(config.vocab_size, config.embed_dim, padding_idx=0)
        self.rnn = nn.LSTM(config.embed_dim, config.hidden_dim, num_layers=config.num_layers, batch_first=True, dropout=config.dropout)
        self.dropout = nn.Dropout(config.dropout)
        self.proj = nn.Linear(config.hidden_dim, config.embed_dim, bias=False)
        self.lm_head = nn.Linear(config.embed_dim, config.vocab_size, bias=False)

    def forward(self, input_ids, labels=None, **kwargs):
        x = self.dropout(self.embedding(input_ids))
        out, _ = self.rnn(x)
        out = self.dropout(out)
        logits = self.lm_head(self.proj(out))
        loss = None
        if labels is not None:
            loss = nn.CrossEntropyLoss(ignore_index=0)(logits.view(-1, logits.size(-1)), labels.view(-1))
        return CausalLMOutput(loss=loss, logits=logits)


tokenizer = AutoTokenizer.from_pretrained("Squeal-Studio/squeal_lstm_5m")
model = LSTMLanguageModel.from_pretrained("Squeal-Studio/squeal_lstm_5m")


def generate(prompt, max_new_tokens=200, temperature=0.9, top_k=50, repetition_penalty=1.05):
    ids = [tokenizer.bos_token_id] + tokenizer.encode(prompt)
    input_ids = torch.tensor([ids])
    with torch.no_grad():
        for _ in range(max_new_tokens):
            logits = model(input_ids).logits[:, -1, :] / temperature
            for token_id in set(input_ids[0].tolist()):
                logits[0, token_id] /= repetition_penalty
            values, _ = torch.topk(logits, top_k)
            logits[logits < values[:, -1:]] = -float("inf")
            next_id = torch.multinomial(torch.softmax(logits, dim=-1), 1)
            if next_id.item() == tokenizer.eos_token_id:
                break
            input_ids = torch.cat([input_ids, next_id], dim=1)
    return tokenizer.decode(input_ids[0].tolist(), skip_special_tokens=True)


print(generate("Однажды в лесу"))

Generation Examples

Prompt: Однажды в лесу

Однажды в лесу я шла только за проживки с маршами! И тогда я была очевидно!
Я отвечаю за нашу природу, где я видел свой фамилии Греции и смоктам все твои
руки гостей повелились, а потом прошли к золотым дяде Весели.

Prompt: Президент заявил

Президент заявил о ситуации на южном строительстве и наступил в суд с юридическим
лицом, переодевшим охрану утопения от украинского "нападения" в проведение акций
протеста, сообщает ТАСС.

Prompt: Наука доказала

Наука доказала, в конце XVIII века в здании, с 1 апреля 1941 по 1952 год.
Всероссийское здание (Войнкот) в город — деревня в Карабахскую сбережь и
присвоено в состав Гракого Сухохенского сельсовета.

Prompt: Linux - это

Linux - это издательная проверка копирования forendispace.ru

Scope & Limitations

• Designed for architectural experiments and educational purposes
• Small vocab (4,000 tokens) limits handling of rare words and emoji
• Context length of 256 tokens limits long-range coherence
• Not intended for production use

License

Apache 2.0

Join us!

Our WebSite: https://squealstudio.ru