Govind222/68m-base-guppy-sft

🐟 Fish LM: 68M Dense Small Language Model (SFT)

Govind222/68m-base-guppy-sft is a high-efficiency 68.96M parameter Dense Small Language Model (SLM) designed for ultra-fast edge inference, local device hosting, and conversational dialog.

Built from scratch using transformer-toolkit, the model was pretrained on 4 Billion tokens of FineWeb and instruction fine-tuned on 57,000 curated dialog examples (arman-bd/guppylm-60k-generic).

• Live Interactive Demo: https://fish-lm.runlog.in/ (hosted entirely on an edge Android smartphone)
• Training Notebook: Fish_LM.ipynb on GitHub
• Live Runlog Telemetry: solar-aurora-298 on Runlog

---

⚡ Highlights & Key Metrics

• Pure Dense Transformer: 100% dense computation with zero MoE overhead, maximizing per-token compute density and caching efficiency on edge CPU/GPU.
• Modern Architecture: Grouped-Query Attention (GQA), SwiGLU feed-forward layers, RoPE rotary positional embeddings, and pre-layer RMSNorm.
• Superb Perplexity: Achieves 1.275 SFT validation perplexity (val_loss: 0.2429) on dialog tasks.
• Extreme Efficiency: Full pretrain + SFT executed for under ~$14 on a single NVIDIA L40S GPU in ~14 hours.
• Mobile-Optimized: Serves real-time tokens at low latency (~35ms TTFT) on mobile ARM big.LITTLE CPUs.

---

📊 Evaluation & Benchmarks

Zero-shot evaluations performed via lm-evaluation-harness demonstrate that Fish LM 68M remains highly competitive with models nearly 2× its size:

Benchmark	Metric	Fish LM (68M Dense)	Pythia 70M	Pythia 160M	GPT-Neo 125M	OPT 125M
PIQA	Accuracy	58.1%	58.6%	61.4%	62.3%	61.8%
Winogrande	Accuracy	48.7%	51.6%	50.4%	49.5%	51.5%
ARC-Easy	Acc Norm	31.4%	34.2%	—	38.9%	39.8%
Overall Average	Mean Acc	39.6%	41.4%	43.7%	43.5%	43.9%

---

⚙️ Model Architecture

Parameter	Value	Specification Notes
Total Parameters	68,957,056 (~68.96M)	Exact non-embedding + embedding tensor weight sum
Architecture Type	Dense Decoder-Only SLM	Pure dense transformer (no MoE routing)
Layers (n_layers)	12	Uniform transformer blocks
Hidden Dim (dim)	512	Model dimension ($d_{\text{model}}$)
Query Heads (n_heads)	8	Head dimension = 64
KV Heads (n_kv_heads)	2	4:1 GQA ratio for compressed KV cache memory
FFN Intermediate Dim	1536	$3 \times d_{\text{model}}$
FFN Activation	swiglu	Dense SwiGLU gating (w1, w2, w3)
Positional Encoding	rope	Rotary Position Embeddings
Normalization	rmsnorm	Pre-norm with $\epsilon = 1\text{e-}6$
Vocabulary Size	32000	Custom Rust BPE Tokenizer
Max Sequence Length	512	Context window length
Weight Tying	false	Untied input embeddings and output LM head

---

📈 Training Details

1. Pretraining

• Base Checkpoint: Govind222/68m-base
• Pretraining Corpus: 4.0 Billion tokens from Hugging Face's FineWeb (sample-10BT)
• Objective: Standard autoregressive next-token prediction with cosine decay schedule

2. Supervised Fine-Tuning (SFT)

• Dataset: 57,000 instruction & conversation pairs from arman-bd/guppylm-60k-generic
• Training Framework: transformer-toolkit
• Telemetry & Tracking: Live monitored on Runlog (Run: solar-aurora-298)
• Hardware: 1× NVIDIA L40S GPU
• Training Throughput: ~6,064 tokens/sec
• Final SFT Validation Loss: 0.2429
• Final SFT Perplexity: 1.275

---

🚀 Quickstart & Inference

Prerequisites

pip install torch transformer-toolkit

Loading and Generating Text

import torch
from transformer_toolkit import TransformerConfig, Transformer, RustBPETokenizer

# 1. Initialize Architecture
config = TransformerConfig(
    vocab_size=32000,
    dim=512,
    n_layers=12,
    n_heads=8,
    n_kv_heads=2,
    ffn="swiglu",
    hidden_dim=1536,
    norm="rmsnorm",
    pos_enc="rope",
    dropout=0.0,
    tie_weights=False,
    max_seq=512,
    use_kv_cache=True,
)

model = Transformer(config)

# 2. Load Weights & Tokenizer
state_dict = torch.load("model.pt", map_location="cpu")
if isinstance(state_dict, dict) and "model" in state_dict:
    state_dict = state_dict["model"]
model.load_state_dict(state_dict)
model.eval()

tokenizer = RustBPETokenizer()
tokenizer.load("tokenizer.json")

# 3. Format Input & Generate
prompt = "<|im_start|>user\nHello! Who are you?<|im_end|>\n<|im_start|>assistant\n"
input_ids = torch.tensor([tokenizer.encode(prompt)], dtype=torch.long)

output_ids = model.generate(
    input_ids,
    max_new=50,
    temperature=0.4,
    top_k=50,
)

response = tokenizer.decode(output_ids[0].tolist())
print(response)

---

📱 Edge & On-Device Deployment

Fish LM was specifically benchmarked and optimized for constrained edge hardware:

• Host Architecture: Android / Termux on ARMv8/ARMv9 big.LITTLE core topology
• Thread Scheduling: Peak throughput achieved on 2 Big CPU cores (OMP_NUM_THREADS=2), preventing thermal throttling and Little-core barrier latency.
• Memory Footprint: Less than ~280 MB RAM at runtime with automatic idle watchdog unloading.

---

📚 Acknowledgments & Citations

• Built and trained with transformer-toolkit by @Barbade22.
• Dialog dataset provided by GuppyLM by Arman (@arman-bd).
• Experiment tracking and live dashboard provided by Runlog.