Densy-330M-SFT (Supervised Fine-Tuned Dense Transformer)

---

⚡ Overview

Densy-330M-SFT is the supervised fine-tuned checkpoint of the pure dense 328.5M parameter causal transformer Densy-330M.

• Base Model: Initialized from Phase-Technologies/densy-330m-pretrain (final / 15.0B tokens).
• Training Objective: Masked cross-entropy across multi-turn user/assistant conversational turns. Loss is computed strictly on assistant response tokens, with prompt tokens masked.
• Data Mixture: Curated multi-turn dialogues spanning coding, math, general instruction following, and structured formatting.
• Compute: Trained across 4× NVIDIA H100 80GB SXM5 GPUs using PyTorch DistributedDataParallel (DDP) with BF16 precision and cosine learning rate decay ($2\times 10^{-5} \to 2\times 10^{-6}$).

Checkpoints

• step_25/
• step_50/
• step_75/
• final/

Chat & Aligned Release

For the fully aligned conversational assistant with Direct Preference Optimization (DPO), see: 👉 Phase-Technologies/Densy-330M-Chat

---

a note from the creator.....

honestly when we set out to build densy we wanted to prove that pure, classic dense transformer architecture still has unmatched elegance and power when pushed to its limits..... after wrestling with complex mixture-of-experts routing and hybrid state space layers in our previous models, densy was our return to first principles: no gate networks, no expert balancing losses, no quadratic memory tricks—just 24 deep, pure dense layers weight-tied to a massive 206k vocabulary running wide open on 4x nvidia h100 sxm5 gpus at over 580,000 tokens every single second.....

watching the loss drop from 12 down across 15 billion tokens in runpod's datacenter was an unbelievable feeling..... training independent frontier-style models with limited personal budgets is terrifying and exhilarating all at once..... every single dollar counted, every step mattered, and when little bugs cropped up in our loss shifts or dataloaders we had to diagnose them in real time on live silicon with the clock ticking.....

huge love and endless gratitude to the runpod team for building the infrastructure that lets independent developers spin up multi-h100 nodes with a single click and actually train real models from scratch..... this journey has been exhausting, beautiful, and deeply humbling..... to everyone following along, experimenting with our models, or building their own architectures in bedrooms and dorms: never let compute monopolies convince you that you can't train from scratch..... i hope you enjoy talking with densy..... thank you for believing in this crazy dream.....

— soham pal (phase technologies)