JohnEnev/modern-llm-v1-base

modern-llm V1 (base, pretrained)

A 353M-parameter decoder-only transformer, built and trained from scratch — no Hugging Face Trainer, raw PyTorch. This is the base pretrained model: strong English fluency, but not instruction-tuned. It completes text; it does not follow instructions or answer questions like a chatbot.

📖 Full writeup: Building a 350M Transformer From Scratch 💻 Code: github.com/JohnEnev/modern-llm

Architecture

Parameters	353,502,208
d_model	1024
Layers	24
Attention heads	16
KV heads	16 (standard MHA, no GQA)
Position encoding	RoPE
Normalization	RMSNorm, Pre-LN
Feed-forward	SwiGLU (8/3 hidden multiplier)
Context length	1024
Vocab size	50,304 (tiktoken GPT-2 BPE, padded)
Optimizer	AdamW
Tied embeddings	Yes

Training

• Data: FineWeb-Edu, 10B tokens (~29 tokens/parameter)
• Hardware: single H100
• Final validation loss: 2.86 (perplexity ≈ 17.5)

How to load

from huggingface_hub import snapshot_download
from safetensors.torch import load_file
import sys

local_dir = snapshot_download("JohnEnev/modern-llm-v1-base")
sys.path.insert(0, local_dir)

from modeling.gpt import GPT, GPTConfig

# NOTE: GPTConfig() defaults are shaped for V2 — override explicitly for V1.
config = GPTConfig(
    vocab_size=50304, d_model=1024, n_layers=24, n_heads=16, n_kv_heads=16,
    max_seq_len=1024, use_flash=True, tie_weights=True,
    use_qk_norm=False, use_diff_attn=False, use_mhc=False, use_xsa=False,
)
model = GPT(config)

state_dict = load_file(f"{local_dir}/model.safetensors")
model.load_state_dict(state_dict, strict=False)  # lm_head re-tied below
model.lm_head.weight = model.token_embeddings.weight
model.eval()

How to sample from it

import torch, tiktoken

enc = tiktoken.get_encoding("gpt2")
input_ids = torch.tensor([enc.encode("The meaning of life is")])
output = model.generate(input_ids, max_new_tokens=50, temperature=0.8, top_k=50)
print(enc.decode(output[0].tolist()))

Related checkpoints

• modern-llm-v1-sft — instruction-tuned via SFT
• modern-llm-v1-grpo — further trained with GRPO

Limitations

Base model, English only, 1024-token context, not instruction-tuned. See the SFT/GRPO checkpoints above for a usable assistant.