AhiskaAI/AhiskaAI-65m-base-v0.1

AhiskaAI LLaMA 65M Alpaca v0.1 (1 Full Epoch Pre-trained)

AhiskaAI LLaMA 65M Alpaca v0.1 is a next-generation, highly optimized Small Language Model (SLM) built completely from scratch for the Turkish language ecosystem.

Unlike older architectures, this model is initialized from zero using the modern LLaMA framework (featuring RoPE, SiLU activation, and RMSNorm). It was pre-trained for 1 full epoch on a 5.3 GB clean Turkish corpus and subsequently instruction fine-tuned via the Alpaca format. It represents a major leap in efficiency, packing modern LLM features into a tight, lightweight ~65M parameter envelope.

---

🧠 Training Methodology & Dataset

To guarantee a structurally sound foundation capable of instruction alignment, the model underwent a rigorous dual-stage training pipeline:

1. Pre-training (1 Full Epoch): Trained on a 5.3 GB filtered Turkish corpus (CulturaX split + Turkish Wikipedia). Running a full epoch allowed the model to genuinely converge on core Turkish syntax, world knowledge, and semantic mappings.
2. Instruction Tuning (SFT): Fine-tuned using a custom Turkish Alpaca matrix to enable prompt compliance, multi-turn dialogue capabilities, and structured Markdown responses.

---

💻 Hardware Constraints & "Fail Forward" Philosophy

At AhıskaAI, we champion transparency and the indie "Build in Public" spirit. This modern LLaMA variant was hammered out entirely on local, consumer-grade hardware:

• Hardware: NVIDIA GeForce RTX 4050 Laptop GPU (6GB VRAM)
• Efficiency: Thanks to its modern LLaMA architecture and ~65M scale, it runs with blazing-fast tokens-per-second on Hugging Face Free CPU Spaces without requiring any premium GPU infrastructure.

---

🛠️ Custom Optimized Turkish Tokenizer

Instead of using English-centric tokenizers that aggressively fragment agglutinative Turkish words, this model deploys a custom BPE Tokenizer with a 32,000 vocab size trained entirely from scratch on our raw corpus.

• Context Efficiency: It natively recognizes Turkish root-suffix patterns, drastically lowering the token-to-word ratio and maximizing the utility of its 1024 context window.

---

📊 Model Architecture & Specifications

As verified by the core configuration, this model abandons legacy frameworks for a pure LLaMA layout:

• Architecture: LlamaForCausalLM
• Parameters: ~65 Million
• Hidden Size (hidden_size): 512
• Intermediate Size (intermediate_size): 1376
• Layers (num_hidden_layers): 12
• Attention Heads (num_attention_heads): 8
• Activation Function: SiLU (silu)
• Context Length (max_position_embeddings): 1024 tokens

---

🛠️ Quickstart Usage (Alpaca Format)

You can easily load and run inference using the Hugging Face transformers library. Make sure to use the exact Alpaca prompt template for the best instruction-following results:

from transformers import LlamaForCausalLM, AutoTokenizer
import torch

model_name = "AhiskaAI/ahiska-llama-65m-alpaca-v0.1"

# Load model and custom tokenizer
model = LlamaForCausalLM.from_pretrained(model_name).to("cuda" if torch.cuda.is_available() else "cpu")
tokenizer = AutoTokenizer.from_pretrained(model_name)

def generate_response(instruction):
    # Standard Alpaca Prompt Template
    prompt = f"Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n{instruction}\n\n### Response:\n"
    
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    
    with torch.no_grad():
        outputs = model.generate(
            **inputs, 
            max_length=200, 
            do_sample=True, 
            top_k=40, 
            top_p=0.92,
            temperature=0.6, # Keeps the small LLaMA focused and realistic
            repetition_penalty=1.15
        )
    
    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return response.split("### Response:\n")[-1].strip()

# Sample Generation
print(generate_response("Sağlıklı yaşamak için 3 ipucu ver"))