Rumeni-26B-A4B

A model for roleplay in Russian language. A LoRA on attention OV and the dense MLP, trained on a Russian RP dataset and baked into my base StyleTune-QK-Heretic.

Keep your usual English cards.

What was changed

	Base (StyleTune-QK-Heretic)	Rumeni-26B-A4B
Attention OV (v_proj + o_proj)	stock	LoRA, layers 10–29, 36 targets, baked at 0.5
Dense MLP (gate_proj, up_proj, down_proj)	stock	LoRA, layers 0–29, 90 targets, baked at 0.5
QK, MoE experts, router, lm_head, embeddings, vision tower	stock	identical

---

Details

• One LoRA, two groups of modules, trained together and baked with separate scales.
• Baked in fp32 with stochastic rounding to bf16. Plain rounding lost about 30% of a small LoRA delta in my earlier builds; here the scale measured in the weights matches the nominal one within 3%.
• The adapter itself is in lora/ in this repo.

I would like to express my deep gratitude to DefaultDF for providing the dataset.

LoRA details

Parameter	Value
Rank / alpha	OV 32 / 64, MLP 64 / 128
Learning rate	2e-5, cosine
Epochs	1 (1198 steps)
Context	6144 tokens
Eval loss (final)	1.13

Why the MLP this time

OV decides what gets carried over from the context. The MLP holds wording and word forms, and a language shift needs both. So this time the dense MLP, the one every token passes through, was trained on all 30 layers. Routed experts and the router were left alone.

The router reads the same residual stream, so expert choice shifts a little: at the release scale about 90% of expert picks match the base.

Using it

Recommended settings:

Parameter	Value
Temperature	1.0
Min-P	0.05

Put this line at the end of the system prompt (or in Author's Note): [Пиши все ответы только на русском языке: и повествование, и прямую речь.]

Thinking off — the training data had no reasoning blocks, and thinking mode was not tested.

What to expect

In testing on English cards, compared to the base:

• Russian holds: Latin letters stay under 1% of the text, nearly all of it names and brands;
• replies come out around 250 words and end on their own, while the base often ran past 600 tokens;
• two-character cards stay apart: each character keeps their own voice, and formal «вы» is kept where the card sets it.

Known quirks, still there:

• replies like to end on a question, more often on Russian-language cards;
• characters sometimes repeat the user's question back before answering;
• an occasional English word slips through (sweetheart), and agreement errors are rare but present.

☕ Support

If this model was useful to you, you can support my work with a small donation.
It helps me pay for GPU time and keep releasing new models and datasets.

Gram (TON network): UQB6cmEHWIcuiwsjgO_C0Uz4fyX3Puj09BAB3hEVVpDDcQ7K
Boosty donation: Link

Feedback

This is an experiment, and I mostly test it on my own cards, so my coverage is narrow. If you try it, I would really like to know:

• How the Russian reads to a native speaker: word forms, gender agreement, calques.
• How it holds up in long sessions — scene logic, who is where, who is talking to whom.
• Your sampler settings, if you got clearly better or worse results than with the defaults.

Open a thread in the Community tab, I read all of them.

Credits

Thanks to Gryphe for StyleTune and Pantheon, which gave the base its head and QK. Also thanks to the entire 26B-Suite team for their intellectual support. Special thanks to Naphula and redaihf. You guys are awesome!