🇮🇳 Cortiqa Falin-300M (Preview)
An early research preview of the sovereign 297M SLM developed by Cortiqa.
Falin-300M is a 297-Million parameter decoder-only transformer model designed, engineered, and trained from scratch by Cortiqa. Built upon the proprietary Menothus architecture, Falin is optimized for extreme low-latency and edge device deployment (consumer GPUs, CPUs, mobile devices, and browser extensions).
🚀 Model Architecture & Innovations
- Total Parameters: 297,034,800 (~300M)
- Extreme Grouped Query Attention (GQA): 16 Query heads to 2 Key-Value heads (8:1 ratio), reducing KV-cache VRAM consumption by 75% during inference.
- Parallel Attention + SwiGLU FFN: Computes Attention and Feed-Forward networks concurrently, improving GPU utilization and decreasing per-layer execution latency.
- Hybrid Sliding Window Attention (SWA): Local context window of 512 tokens with every 4th layer computing dense global causal attention.
- FlashAttention / SDPA Native: Fully optimized for scaled dot product attention.
| Specification | Value |
|---|---|
| Layers | 24 |
| Hidden Dimension | 1024 |
| Query Heads | 16 |
| Key-Value Heads | 2 |
| Intermediate FFN Dim | 2816 (SwiGLU) |
| Max Context Length | 1024 tokens |
| Vocabulary Size | 32,000 (BPE) |
🛠️ How to Run Inference
1. Requirements
pip install torch tokenizers
2. Python Inference Code
import os, json, torch
import torch.nn.functional as F
from huggingface_hub import snapshot_download
# Download model repository from Hugging Face
model_dir = snapshot_download(repo_id="YOUR_HF_USERNAME/falin-300m")
# Load model weights and config
# (Use the Menothus architecture code from the repo)
🏢 About Cortiqa
Falin-300M was designed, engineered, and pre-trained from scratch by Cortiqa, focusing on building sovereign, ultra-fast, and resource-efficient AI architectures for India and the global developer ecosystem.
- Developer: Cortiqa
- Architecture: Menothus
- Release Version: v0.1-Alpha
- Downloads last month
- -