ueban-1.3

A tiny Russian language model with 180M parameters, based on the Llama architecture and designed to run on edge devices.

Specification

Total parameters	180,186,624
Vocabulary size	50,257
Embedding dimensions	768
Hidden layers	20
Attention heads	12
Head dimension	64
Context length	1024

Prompt format

The model works with plain-text instructions in this format:

### Инструкция:
{question}
### Ответ:

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("RaspizdAI/ueban-1.3")
model = AutoModelForCausalLM.from_pretrained("RaspizdAI/ueban-1.3", dtype="bfloat16").cuda().eval()

prompt = "### Инструкция:\nКто ты?\n### Ответ:\n"
ids = tok(prompt, return_tensors="pt").input_ids.cuda()
with torch.no_grad():
    out = model.generate(ids, max_new_tokens=60, do_sample=True, temperature=0.8, top_p=0.9,
                         top_k=50, repetition_penalty=1.15,
                         pad_token_id=tok.pad_token_id, eos_token_id=tok.eos_token_id)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

Loss curve

License & Attribution

This model is licensed under CC-BY-Sa-4.0.

• You are free to share and adapt the material.
• You must give appropriate credit, provide a link to the license, and indicate if changes were made.
• You cannot remove the RaspizdAI copyright and attribution, nor can you close the source of derivative models.