akanwar/delhi-cpt-lora-adapter

🏛️ Delhi Fiction Writing Assistant — Continual Pre-Training (CPT) Adapter (Phase 4)

This checkpoint contains the Continual Pre-Training (CPT) LoRA adapter for sarvamai/sarvam-1 (2.0B SLM), trained on 9.03M tokens of domain-weighted Delhi heritage, urban geography, and literary style priors.

📊 Training Telemetry (NVIDIA A100-SXM4-40GB)

• Epochs: 3 full passes (1,104 optimizer steps)
• Tokens Trained: 9,031,680 tokens
• Effective Batch Size: 16,384 tokens / step (8 sequences $\times$ 2048 sequence packing)
• Starting Loss $\rightarrow$ Final Loss: 4.4339 $\rightarrow$ 1.2609 (-71.6%)
• Starting Perplexity $\rightarrow$ Final Perplexity: 84.26 $\rightarrow$ 3.53 (-95.8%)

⚙️ LoRA Configuration

• Rank ($r$): 64
• Alpha ($\alpha$): 128
• Dropout: 0.05
• Target Modules: All linear attention & MLP projections + embed_tokens & lm_head
• Trainable Parameters: 105.6M (3.97% of network)
• Weights File: adapter_model.safetensors (986 MB)

🚀 Quick Verification

./venv/bin/python3 scripts/12_eval_generation.py --num_prompts 1 --max_new_tokens 50