minimind-135m-cpt

Continued-pretrained MiniMind (135M) base language model.

Model details

• Architecture: decoder-only Transformer (GPT-style), 12 layers / 768 dim / 12 heads, block size 512
• Parameters: ~135M
• Tokenizer: 32k byte-level BPE (tokenizer.json)
• Continued-pretrained from an OpenWebText base on ~2.57B tokens (Wikipedia / Books / FineWeb-Edu / C4), ~1 epoch.
• Val perplexity ~30.9 on the held-out mix.

Files

• minimind_model.pt — native PyTorch checkpoint (dict with model_state_dict + config)
• tokenizer.json / tokenizer_config.json / special_tokens_map.json

Load (native)

import torch
from <your_module> import MiniMind   # class definition lives in the training notebook
ck = torch.load('minimind_model.pt', map_location='cpu')
model = MiniMind(**ck['config'])
model.load_state_dict(ck['model_state_dict'])