SeightCZE/ABA-1.0.2-0.12B

ABA-1 (124M) — Conversational & Code-Generation Language Model

ABA-1 is a custom Decoder-only Transformer model (LLaMA-style architecture) with 124.3 million parameters, trained entirely from scratch on a single consumer GPU — an NVIDIA GeForce RTX 3050 Laptop GPU (4 GB VRAM).

The model is natively bilingual (Czech & English) and specialises in programming fundamentals across 6 languages: Python, C, C#, JavaScript, HTML5, and CSS3.

---

🚀 Model Architecture

Parameter	Value
Parameters	124,280,832 (~124.3M)
Type	Autoregressive Decoder-only Transformer
Embedding dimension (dim)	768
Layers (n_layers)	12
Attention heads (n_heads)	12 (64 dim per head)
MLP hidden size (d_ff)	2,560 (SwiGLU activation)
Normalisation	RMSNorm (eps = 1e-6)
Positional encoding	Rotary Position Embeddings (RoPE, θ = 10,000)
Attention	FlashAttention / PyTorch SDPA
Context length (max_seq_len)	512 tokens
Vocabulary size	16,384 tokens (custom Byte-Level BPE)

---

📊 Training History

1. Pre-training (from scratch)

   • 8.12 million tokens of code and bilingual text.
   • 3,000 training steps with cosine LR schedule and FP16 AMP.
   • Final validation loss: 1.0367 (Perplexity: 2.82, Top-1 accuracy: 90.56%).

2. Supervised Fine-Tuning (SFT — Chat Assistant)

   • Dialogue-boundary-aligned padded batching with loss masking (ignore_index=-100 on prompts and padding).
   • 2,149 targeted bilingual technical dialogues (Bactrian-X CS, Stanford Alpaca EN, curated algorithm & identity data).
   • Checkpoint: aba_ai_chat_best.pt (Step 200, Val Loss: 1.9384).

---

💻 Quick Start (Python)

No heavy external libraries are required — just torch:

import torch
from model import AbaConfig, AbaTransformer
from aba_tokenizer import AbaTokenizer

# 1. Load config and model
config = AbaConfig(
    vocab_size=16384,
    max_seq_len=512,
    dim=768,
    n_layers=12,
    n_heads=12,
    d_ff=2560
)
model = AbaTransformer(config)

# Load weights (safetensors preferred, falls back to .bin)
try:
    from safetensors.torch import load_file
    weights = load_file("model.safetensors")
except ImportError:
    weights = torch.load("pytorch_model.bin", map_location="cpu")

model.load_state_dict(weights)
model.eval()

# 2. Tokenizer
tokenizer = AbaTokenizer()

# 3. Format the prompt
prompt = "<|user|>\npython bubble sort\n<|assistant|>\n"
idx = torch.tensor([tokenizer.encode(prompt)], dtype=torch.long)

# 4. Generate a response
with torch.no_grad():
    out_ids = model.generate(idx, max_new_tokens=200, temperature=0.25)

response = tokenizer.decode(out_ids[0].tolist(), skip_special_tokens=True)
print(response)

---

🏷️ Special Conversation Tokens

Token	ID	Purpose
<|endoftext|>	0	End of sequence / document
<|user|>	—	Start of a user message
<|assistant|>	—	Start of an assistant reply

---

📜 License

MIT License — free for educational, research, and commercial use. Built as part of the ABA AI project.