project-828-gpt-base

A 398.7M parameter Mixture-of-Experts (MoE) causal language model optimized for code generation and technical reasoning, pretrained on ~60B tokens.

398.7M total parameters, ~286M active per token — 4 routed experts + 1 shared expert with top-2 routing per token.

Model Architecture

Property	Value
Architecture	Decoder-only Transformer with MoE FFN layers
Total Parameters	398.7M
Active Parameters / Token	~286M
Hidden Dimension ($d_{\text{model}}$)	768
Intermediate Size ($d_{\text{ff}}$)	760
Hidden Layers	24
Attention Heads / KV Heads	12 / 6 (GQA 2:1)
Head Dimension	64
Routed Experts	4
Active Experts / Token	2 (Top-2)
Shared Experts	1
Context Length	2048 (extensible to 8192 via YaRN)
Vocabulary Size	49152
Precision	BFloat16 Mixed Precision
Positional Encoding	RoPE with YaRN scaling support

Parameter Breakdown

Component	Parameters
Embeddings	~37.7M
Unembedding	~37.8M
Attention (×24 layers)	~1.8M each
MoE FFN (×24 layers)	~11.7M each
Layer Norms + Misc	~0.07M
Total	398.7M

Key Architectural Features

• Grouped Query Attention (GQA): 12 query heads with 6 key-value heads (2:1 ratio) for memory-efficient attention
• QK-Norm: RMSNorm applied to query and key projections before RoPE for attention stability
• Auxiliary-Loss-Free MoE Routing: Sigmoid gating with dynamic bias adjustment (DeepSeek-V3 paper) — achieves near-perfect ~25% utilization per expert without auxiliary losses
• SwiGLU Activation with Soft-Clamping: limit=7.0 prevents activation explosions during long training runs
• Batched Expert Dispatch: Sort-and-slice dispatch with searchsorted boundaries for contiguous memory access
• RoPE with YaRN Extension: Base context of 2048 tokens, extensible to 8192 via YaRN scaling

Training Details

Phase 1 — Pretraining (~60B tokens)

Training Config	Value
Hardware	H200 GPU
Peak Learning Rate	3e-4
Min Learning Rate	3e-5
Scheduler	WSD (Warmup-Stable-Decay)
Warmup Steps	500
Total Steps	101,726
Effective Batch Size	37 × 8 = 296 sequences
Tokens per Step	~0.61M
Gradient Clipping	1.0

Training Data Mix

Dataset	Weight	Category
starcoderdata — Python	14	Source Code
starcoderdata — JavaScript	8	Source Code
starcoderdata — Java	6	Source Code
starcoderdata — TypeScript	4	Source Code
starcoderdata — C++	6	Source Code
starcoderdata — C	4	Source Code
starcoderdata — C#	3	Source Code
starcoderdata — Go	4	Source Code
starcoderdata — Rust	3	Source Code
starcoderdata — PHP	3	Source Code
fineweb-edu-dedup	20	General Knowledge
cosmopedia-v2	7	General Knowledge
wikipedia-en	3	General Knowledge
finemath-4plus	8	Math / Reasoning
stackexchange (programming/CS)	7	CS / Engineering

Category breakdown: Source Code 55% · General Knowledge 30% · Math/Reasoning 8% · CS/Engineering 7%

Quickstart

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "AkshithAI/project-828-gpt-base"

tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    torch_dtype="bfloat16",
    device_map="auto",
    trust_remote_code=True
)

prompt = "def binary_search(arr, target):"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=200,
    temperature=0.7,
    top_p=0.9,
    do_sample=True,
    use_cache=False,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation

@misc{project828,
  author = {AkshithAI},
  title = {Project 828: MoE Transformer with Training Pipeline},
  year = {2025},
  publisher = {GitHub},
  url = {https://github.com/AkshithAI/project_828}
}