SeightCZE/ABA1-1.1.0-0.12B

ABA-1 (124M) — Czech and English Conversational Code Model

ABA-1 is a decoder-only Transformer trained from scratch. It is intended for Czech and English conversation and programming fundamentals across Python, C, C#, JavaScript, HTML, and CSS.

Model details

Property	Value
Parameters	124,275,456
Hidden size	768
Layers	12
Attention heads	12
MLP size	2560
Context window	512 tokens
Vocabulary	16384 tokens
Architecture	Causal decoder, RoPE, RMSNorm, SwiGLU, PyTorch SDPA

Training snapshot

• Pretraining: 3,000 steps on an 8.12 million-token Czech, English, and programming corpus. Recorded validation loss: 1.0367.
• Chat fine-tuning: checkpoint aba_ai_chat_best.pt, step 500, validation loss 0.8775.
• The local dataset snapshot contained 4,554 dialogues (4,098 training and 456 validation).
• Hardware used for training: NVIDIA GeForce RTX 3050 Laptop GPU with 4 GB VRAM.

Use with a Transformers pipeline

Install the runtime packages:

pip install torch "transformers>=4.50.0" tokenizers safetensors

The model implementation is custom code, so trust_remote_code=True is required when loading the model. The tokenizer uses the standard PreTrainedTokenizerFast wrapper over the included tokenizer.json.

from transformers import pipeline

repo_id = "SeightCZE/ABA1-1.1.0-0.12B"
generator = pipeline(
    "text-generation",
    model=repo_id,
    tokenizer=repo_id,
    trust_remote_code=True,
)

prompt = "<|user|>\nHello, who are you?\n<|assistant|>\n"
result = generator(
    prompt,
    max_new_tokens=128,
    do_sample=True,
    temperature=0.25,
    top_p=0.9,
)
print(result[0]["generated_text"])

Use with AutoTokenizer and AutoModelForCausalLM

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "SeightCZE/ABA1-1.1.0-0.12B"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
)
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

prompt = "<|user|>\nWrite a Python function that checks whether a score is above 50.\n<|assistant|>\n"
inputs = tokenizer(prompt, return_tensors="pt").to(device)
with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=160,
        do_sample=True,
        temperature=0.25,
        top_p=0.9,
    )
new_tokens = output_ids[0, inputs["input_ids"].shape[1] :]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

The model uses a 512-token sliding context window and does not use a KV cache. Generation is compatible with the standard Transformers generate() API, but will be slower than architectures with cached attention.

Run the included chat program

On Windows, install the dependencies with install_hf.bat, then start run_chat_hf.bat. The launcher loads the exported model through AutoModelForCausalLM and the tokenizer through AutoTokenizer.

Files

• model.safetensors — exported model weights.
• config.json — model configuration and AutoClass mapping.
• generation_config.json — recommended generation defaults.
• configuration_aba.py, modeling_aba.py — custom Transformers model implementation.
• tokenizer.json, tokenizer_config.json — tokenizer data and Transformers metadata.
• chat_hf.py, run_chat_hf.bat, install_hf.bat — standalone chat and setup scripts.
• tokenizer.json, tokenizer_config.json — serialized fast tokenizer and Transformers metadata; AutoTokenizer loads the tokenizer from tokenizer.json.
• vocab.json, merges.txt — standalone BPE vocabulary and merge rules matching the data bundled in tokenizer.json (16,384 tokens and 16,116 merge rules).
• LICENSE — project license.

License

MIT License. See LICENSE.