Phase-Technologies/hensy-330m-pretrain

Hensy-330M (5.0 Billion Token Foundation Model)

---

⚡ Overview

Hensy-330M is an open-source, pure dense causal transformer foundation model pretrained from raw scratch across 5.00 Billion tokens on 4× NVIDIA H100 80GB SXM5 GPUs connected via NVLink on RunPod.

Architecture Specifications

• Parameters: 328,508,928 (328.5M)
• Layers: 24 Pre-LayerNorm Transformer blocks
• Hidden Dimension ($d_{\text{model}}$): 768
• Attention Heads: 12 heads (64-dim per head) with RoPE and SDPA
• MLP Dimension: 3,072 ($4 \times d_{\text{model}}$) with GeLU
• Context Length: 4,096 tokens
• Vocabulary Size: 206,464 tokens (CRAYON Turbo)
• Weight Tying: Input embedding tied to LM output projection head

Checkpoints

• step_500/ (2.10B tokens)
• step_1000/ (4.19B tokens)
• final/ (5.00B tokens complete)

Downstream SFT Model

👉 Phase-Technologies/hensy-330m-sft

---

🚀 Readymade 1-Click Google Colab Code (Raw Completion)

Paste this single cell directly into Google Colab:

# 1. Install dependencies
!pip install -q torch huggingface_hub xerv-crayon safetensors

import sys
import torch
from pathlib import Path
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

REPO_ID = "Phase-Technologies/hensy-330m-pretrain"
SFT_REPO = "Phase-Technologies/hensy-330m-sft"
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
DTYPE = torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float32

# Download model architecture & weights
for f in ["config.py", "model.py"]:
    if not Path(f).exists():
        hf_hub_download(repo_id=SFT_REPO, filename=f, local_dir=".")

cfg_json = hf_hub_download(repo_id=REPO_ID, filename="final/model_config.json")
st_path = hf_hub_download(repo_id=REPO_ID, filename="final/model.safetensors")

sys.path.insert(0, ".")
from config import HensyConfig
from model import build_hensy_330m
from tokenizer.crayon_wrapper import ForgeTokenizer

tokenizer = ForgeTokenizer()
cfg = HensyConfig.from_json(cfg_json)
model = build_hensy_330m(cfg)

weights = load_file(st_path)
weights = {k.replace("_orig_mod.", ""): v.to(DTYPE) for k, v in weights.items()}
model.load_state_dict(weights, strict=False)
model = model.to(DEVICE).to(DTYPE).eval()
print("✅ Hensy-330M Base loaded cleanly!\n")

def complete(prompt: str, max_new_tokens: int = 64, temperature: float = 0.7):
    input_ids = torch.tensor([tokenizer.encode(prompt, add_bos=False, add_eos=False)], device=DEVICE)
    with torch.inference_mode():
        out = model.generate(input_ids, max_new_tokens=max_new_tokens, temperature=temperature, repetition_penalty=1.15)
    return tokenizer.decode(out[0].tolist(), skip_special_tokens=False)

print(complete("In physics, gravity is"))

---

a note from the creator.....

building hensy was our redemption arc and our return to discipline..... after uncovering the identity copy shortcut in our earlier experiments, we tore the training loop down to bare metal, fixed every single index shift and dtype promotion from first principles, and spun up 4x nvidia h100 sxm5s to run a clean, uncompromised 5 billion token pretrain from raw scratch.....

watching the loss drop smoothly from 12.4 down to 4.3 over 5 billion tokens at more than 565,000 tokens every second was the most rewarding sight of this entire journey..... following pretrain immediately with supervised fine-tuning cemented the turn-taking and conversational capability without needing heavy alignment crutches.....

endless gratitude and huge shoutout to the runpod.io team for providing the world-class infrastructure, blazing nvlink fabric, and uninterrupted stability that makes training real foundation models possible for independent builders on a personal budget..... to everyone reading this: build from first principles, check your loss gradients on live silicon, and keep pushing open source forward.....

— soham pal (phase technologies)