🧠 MiniMind Course Checkpoints (boyuyang/minimind-course-weights)

This repository hosts the pre-trained and supervised fine-tuned (SFT) PyTorch model checkpoints for our hands-on LLM course built on MiniMind.

All checkpoints are stored under the out/ directory so they can be downloaded directly into /content/minimind in Google Colab and loaded seamlessly by the training and evaluation scripts.

---

📦 Checkpoint Inventory & Experiment Specifications

File Path	Training Stage	Hidden Size (--hidden_size)	Layers (--num_hidden_layers)	Max Seq Len (--max_seq_len)	Parameters	Target Hardware & Experiment Description
out/pretrain_128.pth	Stage 1: Pre-training (train_pretrain.py)	128	4	128	1.71M	Ultra-Lightweight CPU/GPU Sanity Check: Trained for 3 epochs (batch_size=16, lr=1e-3) on a tiny 128-sample subset to verify the training loop in < 5 seconds even on CPU.
out/full_sft_128.pth	Stage 2: Full SFT (train_full_sft.py)	128	4	128	1.71M	Ultra-Lightweight SFT Sanity Check: Initialized from pretrain_128.pth and fine-tuned for 3 epochs (batch_size=16, lr=5e-4) to validate the end-to-end SFT & inference pipeline.
out/pretrain_256.pth	Stage 1: Pre-training (train_pretrain.py)	256	6	128	6.66M	GPU Hands-On Pre-training (Lesson 1 Default): Trained on an NVIDIA A100 GPU for 6 epochs (batch_size=32, lr=1e-3) over 1,200 English passages (pretrain_mini_test.jsonl), converging from loss 5.98 to 0.027.
out/full_sft_256.pth	Stage 2: Full SFT (train_full_sft.py)	256	6	128	6.66M	GPU Hands-On Instruction Tuning (Lesson 1 Default): Initialized from pretrain_256.pth and fine-tuned on an NVIDIA A100 GPU for 8 epochs (batch_size=32, lr=5e-4) over 1,200 English Q&A pairs (sft_mini_test.jsonl), converging to loss 0.014 for fluent English question answering.

---

🚀 Quickstart: Downloading & Loading Checkpoints in Colab

1. Download Checkpoints via huggingface_hub

from huggingface_hub import snapshot_download

# Downloads the `out/` folder directly into /content/minimind/out
snapshot_download(
    repo_id="boyuyang/minimind-course-weights",
    repo_type="model",
    local_dir="/content/minimind",
)

2. Run Inference with the 6.66M English SFT Checkpoint (out/full_sft_256.pth)

Note: When instantiating MiniMindConfig, ensure hidden_size and num_hidden_layers match the checkpoint suffix (128 $\rightarrow$ hidden_size=128, num_hidden_layers=4; 256 $\rightarrow$ hidden_size=256, num_hidden_layers=6).

import torch
from transformers import AutoTokenizer
from model.model_minimind import MiniMindConfig, MiniMindForCausalLM

device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained("/content/minimind/model")

# Match the architecture of full_sft_256.pth (6.66M parameters)
config = MiniMindConfig(hidden_size=256, num_hidden_layers=6, max_seq_len=128)
model = MiniMindForCausalLM(config).to(device)

state_dict = torch.load("/content/minimind/out/full_sft_256.pth", map_location=device)
model.load_state_dict(state_dict, strict=False)
model.eval()

prompt = "What is artificial intelligence?"
messages = [{"role": "user", "content": prompt}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(device)

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=64, do_sample=False)

response = tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(f"Prompt: {prompt}\nResponse: {response}")

---

🔗 Companion Course Repositories

• Course Dataset Repository: boyuyang/minimind_dataset
• Full-Scale Reference Models: boyuyang/MiniMind2 & boyuyang/MiniMind2-Small