Abiray/Kshana-170M-Base

🌌 Kshana-170M Base

Kshana-170M-Base is a compact 170M-parameter foundational causal language model built by Abiray. Moving along the architectural lineage of its predecessor (Sutra), Kshana is trained from scratch using a highly optimized Llama-style architecture with Grouped-Query Attention (GQA) for blazing inference velocity.

Despite its compact size, it achieves highly competitive results on key reasoning benchmarks, making it an optimal base for downstream fine-tuning workflows or resource-constrained edge deployment.

Note: As a raw base model, it requires downstream instruction tuning to perform as a conversational chat agent.

🏆 Benchmarks

The base weights were evaluated head-to-head against sub-500M architectures using lm-evaluation-harness within an identical runtime environment. To align with open-source presentation standards, scores reflect peak performance metric selection targets (acc for science and single-token knowledge choice selections, acc_norm for length-penalized situational context completions).

Benchmark	🌌 Kshana-170M (Ours)	🪵 SmolLM2-135M	🌾 Nandi-Mini-150M	📐 Pythia-160m	🔹 OPT-125m	🧮 Cerebras-256M	⚙️ Pythia-410m
Parameters	169.9M	135M	150M	160M	125M	256M	410M
SciQ (Sci)	81.90%	84.10%	89.10%	55.70%	78.20%	75.70%	80.40%
PIQA (Logic)	66.81%	68.34%	65.13%	59.19%	62.62%	61.10%	66.70%
ARC-Easy (Know)	57.07%	64.39%	54.67%	37.58%	42.76%	40.99%	51.98%
HellaSwag (Ctx)	39.84%	43.17%	37.11%	30.49%	31.62%	28.60%	40.02%

🧠 Model Architecture

Kshana-170M is based on the LlamaForCausalLM architecture with a native Grouped-Query Attention (GQA) layout to compress hardware footprint:

Parameter	Value
Parameters	169,906,752
Hidden size	576
Layers	32
Attention heads	9
KV heads (GQA)	3
Head dimension	64
Intermediate size	1,536
Activation	SwiGLU (silu)
Max Context	8,192 tokens
Vocabulary size	49,152

⚙️ Training Configuration

Parameter	Value
Optimizer	AdamW
Learning rate	3e-4
LR scheduler	Cosine Decay
Precision	bfloat16 / float16 hybrid

📚 Training Data

Trained on a volume of 65 Billion tokens. The corpus characteristics include high-quality deduplicated web extracts, structured synthetic reasoning texts, and educational literature subsets (focusing on FineWeb-Edu, Wikipedia, and Cosmopedia). Data was rigorously filtered using MinHash LSH deduplication and language filtering matrices.

🎯 Operational Scope & Intended Use

✅ Targeted Applications

• Downstream Fine-Tuning (SFT/DPO): Acts as a clean, lightweight base for training specialized assistants, custom chat agents, or task-specific models.
• Local & Edge Deployment: Designed with Grouped-Query Attention (GQA) for efficient quantization (via llama.cpp / GGUF), making it ideal for low-power hardware like consumer CPUs, laptops, and mobile devices.
• Text Completion & Routing: Well-suited for low-latency text continuation, basic autocomplete features, or classification tasks like routing user queries quickly before passing them to larger models.

❌ Out-of-Scope Limits

• Coding & Mathematics: The model's training data consists strictly of natural language text (FineWeb-Edu and Cosmopedia). Because it was never exposed to structured math datasets or code repositories during training, it cannot write code scripts, debug software, or calculate mathematical formulas.
• Factual Knowledge Retrieval: Trained on a strict budget of 65 Billion tokens with a sub-200M parameter boundary, the model lacks the capacity to serve as an open-domain factual encyclopedia. It will hallucinate facts if asked about niche topics without being provided reference text directly in the prompt (e.g., via RAG).
• Interactive Chat (Out of the box): As a raw base model, it will naturally attempt to autocomplete text rather than hold a conversational dialogue. It requires standard instruction fine-tuning before it can be used as a traditional chatbot.

🚀 Inference & Edge Deployment

The model can be initialized within minutes using standard workflows via the Hugging Face transformers environment. Its native GQA layout makes it highly compatible with quantization layers (via llama.cpp / GGUF) to run on consumer CPUs or embedded devices at extreme tokens-per-second.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Abiray/Kshana-170M-Base"

# Initialize matching vocabulary tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id, token=True)

# Pull weights matching verified float16 layout
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto",
    token=True
)

prompt = "The basic physical principle behind gravitational collapse is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs, 
        max_new_tokens=64,
        temperature=0.6,
        top_p=0.85,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))