FORGE-500M-Base

FORGE-500M-Base is the foundational base pretrained checkpoint of the FORGE (First-principles Optimal Recurrent-Gated Expert) architecture, developed by Phase Technologies.

This checkpoint represents the raw pretrained base model after 821 optimizer steps (~1.66 Billion pretraining tokens) across 4× NVIDIA B300 SXM6 MIG compute slices on runpod.io.

For the conversational post-DPO aligned version, see Phase-Technologies/FORGE-500M-Chat.

---

Architecture Overview

Parameter	Value	Details
Total Parameters	500M (0.500B)	Full parameter footprint
Active Parameters	320M (0.320B)	Parameters engaged per forward token pass
Layers	18 total	3 MHA (Attention) + 15 ARG (Adaptive Recurrent Gated SSM)
Feed-Forward Layers	18 total	9 HSE (Hierarchical Sparse Experts) + 9 Dense SwiGLU
SSM Hidden State	64	State dimension for O(1) inference recurrence
Hidden Dimension	1,024	Model width ($d_{\text{model}}$)
Vocabulary Size	206,464	Optimized multi-lingual and code vocabulary
Context Length	4,096 tokens	Pretrained sequence length window

---

How to Run Inference

1. Google Colab (1-Click Quickstart)

In a Google Colab notebook (GPU runtime), paste and run this single cell:

# 1. Clone repository & enter directory
!git clone https://huggingface.co/Phase-Technologies/forge-500m-base
%cd forge-500m-base

# 2. Install lightweight dependencies
!pip install -q torch einops triton transformers huggingface_hub

# 3. Generate text immediately
!python run_inference.py --prompt "The future of artificial intelligence is" --max_new_tokens 128

---

2. Pure Python API (Colab / Jupyter / Local)

import sys
import torch
from pathlib import Path
from huggingface_hub import snapshot_download

# 1. Download model repository
repo_dir = Path(snapshot_download(repo_id="Phase-Technologies/forge-500m-base"))
sys.path.insert(0, str(repo_dir))

# 2. Load tokenizer and model config
from tokenizer.crayon_wrapper import ForgeTokenizer
from config import ForgeModelConfig
from model.forge_model import build_forge_500m

tokenizer = ForgeTokenizer.from_pretrained(str(repo_dir / "tokenizer"))
config = ForgeModelConfig.from_json(str(repo_dir / "model_config.json"))
model = build_forge_500m(config)

# 3. Load BF16 weights
weights = torch.load(repo_dir / "pytorch_model.bin", map_location="cpu")
model.load_state_dict({k: v.to(torch.bfloat16) for k, v in weights.items()})
model = model.to("cuda" if torch.cuda.is_available() else "cpu").to(torch.bfloat16)
model.eval()

# 4. Prompt & Autoregressive Generation
prompt = "In the heart of the ancient forest,"
input_ids = torch.tensor([tokenizer.encode(prompt, add_bos=True, add_eos=False)], device=model.device)

with torch.inference_mode():
    output_ids = model.generate(
        input_ids=input_ids,
        max_new_tokens=100,
        temperature=0.7,
        top_p=0.9,
    )

print(tokenizer.decode(output_ids[0].tolist(), skip_special_tokens=False))

---

Training Logs

The complete historical training log documenting the entire pretraining loss curve and hardware metrics from Step 1 to Step 821 is archived directly in this repository as training_journey.log.

---

a note from the creator.....

honestly i really wanted to train this beauty for full 15b tokens..... that was the real plan from day one to see how far this custom architecture could push..... but running multiple top tier gpus burns money so fast and i just ran completely out of budget..... literally had to stop pretraining early at 821 steps..... huge shoutout and love to runpod.io for providing the insane compute and infrastructure that made training this beast possible..... every single cent went into this little 500m model and seeing it come alive from scratch with custom cuda kernels and our own tokenizer was pure magic..... maybe one day with better funding we will train the full 3b or 15b token run..... until then i hope you guys enjoy playing with this model as much as i loved building it..... thank you so much to everyone who supported this crazy journey.....