MiniMind-135M

A 135M parameter decoder-only transformer (GPT-2 style) pretrained on OpenWebText from scratch.

Model Details

Attribute	Value
Parameters	134.6M
Vocab size	32,000
Hidden dim	768
Layers	12
Attention heads	12
Context window	512
Training data	OpenWebText (~615M tokens)
Training steps	100,000
Best val loss	4.0174
Tokenizer	Custom ByteLevel BPE (32k)

Usage

from transformers import AutoTokenizer
import torch

# Load tokenizer
tok = AutoTokenizer.from_pretrained("achavan1211/minimind-135m")

# Load model (requires MiniMind class definition)
model = MiniMind(32000, 768, 12, 12, 512, 0.1)
state_dict = torch.load("best_model.pt", map_location="cpu")
model.load_state_dict(state_dict["model_state_dict"])
model.eval()

# Generate
prompt = "The future of artificial intelligence"
ids = tok.encode(prompt, return_tensors="pt")
with torch.no_grad():
    out = model.generate(ids, max_new_tokens=100, temperature=0.8)
print(tok.decode(out[0], skip_special_tokens=True))

Training

• Optimizer: AdamW (lr=3e-4, betas=(0.9, 0.95), weight_decay=0.1)
• Schedule: Warmup (2k steps) + cosine decay to 3e-5
• Effective batch: 48 (12 × 4 gradient accumulation)
• Hardware: A100 GPU
• Duration: ~2.5 days

Limitations

This is a base model (no instruction fine-tuning). It generates text continuations, not conversational responses. For assistant-like behavior, it needs further fine-tuning on instruction data.