808 AI (150M) - Custom Lightweight Base Architecture
808 AI is an ultralight, edge-optimized, 150-million-parameter Causal Language Model designed and built from scratch using PyTorch. Designed for high-speed inference, ultra-low latency, and minimal memory usage, this architecture combines state-of-the-art LLM innovations into a compact parameter budget.
Architected & Developed by: Sami
🌟 Architectural Highlights
- Grouped-Query Attention (GQA): 3:1 Query-to-Key/Value ratio for fast auto-regressive generation and minimal KV-Cache footprint.
- Sliding Window Attention (SWA): Efficient contextual representation handling longer input sequences.
- QK-Normalization (QK-Norm): Prevents attention score exploding/vanishing and ensures numerical stability during FP16/BF16 training.
- SwiGLU Activation: Gated MLP structure for higher capacity and richer feature representation.
- Tied Embeddings: Shared weight matrices between input token embeddings and output LM Head for maximum parameter efficiency.
📊 Model Specifications
| Parameter | Value |
|---|---|
| Creator / Developer | Sami |
| Total Parameters | ~150M |
| Hidden Size ($d_{model}$) | 768 |
| Layers | 20 |
| Attention Heads | 12 |
| GQA Ratio | 3:1 |
| Activation | SwiGLU |
| Target Hardware | Edge Devices, Single GPUs, CPU-Inference |
💻 Quickstart & Usage
You can instantiate and run this custom architecture using Hugging Face's transformers library with trust_remote_code=True:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Sadatsami/808-AI-150M-Base"
# Load Model & Tokenizer
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
torch_dtype=torch.float16
)
print("✅ 808 AI Architecture initialized successfully!")
print(f"Total Parameters: {sum(p.numel() for p in model.parameters()):,}")
@misc{sami2026808ai,
author = {Sami},
title = {808 AI: Ultralight 150M Custom Causal Language Model},
year = {2026},
publisher = {Hugging Face},
journal = {Hugging Face Model Hub}
}