achavan1211/minimind-instruct-135m

MiniMind-Instruct-137.7M

A ~137.7M-parameter GPT-style decoder transformer built entirely from scratch - custom architecture, custom tokenizer, from-scratch pretraining, and instruction fine-tuning. This is the instruction-tuned checkpoint: it follows the Alpaca prompt format and responds to instructions.

Educational toy model. Trained on a small subset for a book project. It is not intended to be useful or factual. See Limitations.

Model details

Architecture	GPT-style decoder (pre-norm, GELU FFN, absolute positional embeddings)
Parameters	137.7M
Layers / Heads / d_model	12 / 12 / 768
Vocabulary	32,000 (custom BPE, bundled)
Context length	512 tokens
Checkpoint	best validation loss, step 1000

Training

Stage 1 - Pretraining (from scratch). Next-token prediction on an OpenWebText subset, ~100k steps. Base checkpoint: best_model.pt.

Stage 2 - Instruction fine-tuning (SFT). Alpaca (52k) + Dolly 2.0 (15k) = ~67k examples. Loss is masked to response tokens only; learning rate 5e-5 (10x lower than pretraining).

	Pretraining	SFT
Objective	Next-token	Response tokens only (prompt masked with -100)
Learning rate	3e-4	5e-5
Steps	~100,000	~1,000-4,500 (early-stopped)

Best validation loss: see training curves.

How to use

This is a custom torch.nn.Module, not a HuggingFace PreTrainedModel, so it is not AutoModelForCausalLM-loadable. Load it via the bundled model.py:

# pip install torch transformers
import torch
from transformers import AutoTokenizer
from model import load_model

REPO = "achavan1211/minimind-instruct-135m"
model, cfg = load_model("config.json", "model.pt", device="cpu")   # files from this repo
tok = AutoTokenizer.from_pretrained(REPO)

PROMPT = ("Below is an instruction that describes a task. "
          "Write a response that appropriately completes the request.\n\n"
          "### Instruction:\n{instruction}\n\n### Response:\n")

def ask(instruction, max_new_tokens=150):
    p = PROMPT.format(instruction=instruction)
    ids = tok.encode(p, return_tensors="pt")
    out = model.generate(ids, max_new_tokens=max_new_tokens, temperature=0.7,
                         top_p=0.9, repetition_penalty=1.2,
                         eos_id=tok.eos_token_id, stop_on_eos=True)
    return tok.decode(out[0], skip_special_tokens=True)[len(p):].strip()

print(ask("What is the capital of France?"))
# -> the capital of france is paris.

Example outputs

Q: Write a short poem about the ocean.

the sea was filled with life, like a river of stars and clouds; but the water had no shore, a sight so gentle and peaceful ...

Q: Give three tips for staying healthy.

1. get enough sleep to keep your body functioning properly.
2. be aware of your physical activity and take breaks throughout the day.
3. get plenty of sleep and focus on your health.

Limitations

• Low factual accuracy - the model lacks capacity for reliable world knowledge; it will hallucinate confidently.
• No multi-step reasoning - it cannot do complex logic at ~137.7M parameters.
• Best at format tasks - short Q&A, reformatting, light creative writing within its training distribution.
• Do not use for anything that matters.

Training data & license

Data	Use	License / terms
OpenWebText (subset)	Pretraining	open
Alpaca (Stanford, 52k)	SFT	intended for research use
Dolly 2.0 (Databricks, 15k)	SFT	Apache-2.0

Because fine-tuning used Alpaca (research-use), this model is released under CC-BY-NC-4.0 (non-commercial). See LICENSE.