GautierP/MiniLLM-40M

Mini-LLM Assistant 40M

A ~40M parameter conversational assistant trained entirely from scratch with PyTorch — no pretrained weights, no distillation. Tokenizer, pretraining and supervised fine-tuning all belong to the Mini-LLM project.

This repository hosts both the SFT assistant (repository root) and the base model it was fine-tuned from (base/).

Model details

Parameters	42,476,032 (input embedding shared with the output head)
Architecture	Llama-style decoder: RMSNorm, RoPE, causal attention, SwiGLU
Context length	1,024 tokens — twice the 512 of the older 34M/70M/150M family runs
Vocabulary	32,768 (BPE)
Pretraining	4.60B tokens (FineWeb-Edu + Wikipedia), 35,096 steps (115 tokens/param) — base val loss 3.202
Fine-tuning	2 epochs of SFT on Dolly-15k, No Robots, OpenAssistant oasst2, alpaca-cleaned and identity/small-talk examples — val loss 3.058
Efficiency	~115 tokens/param — zero-shot scores at or above the family 70M (1.44B tokens) with 40% fewer parameters

Base model

The base/ subfolder contains the pretrained checkpoint (mini-llm-base-40M-35096steps.pt) before SFT: 42,476,032 parameters, 32,768-token BPE vocabulary, ~4.60B tokens (FineWeb-Edu + Wikipedia), 35,096 steps — base val loss 3.202. It is a raw completion model, not an assistant: base/ ships without a chat template and without the assistant's sampling defaults.

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "GautierP/MiniLLM-40M"
tok = AutoTokenizer.from_pretrained(repo, subfolder="base", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, subfolder="base", trust_remote_code=True)

inputs = tok("The capital of France is", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=64, do_sample=True, temperature=0.8, top_k=50)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "GautierP/MiniLLM-40M"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)

messages = [{"role": "user", "content": "What is the capital of France?"}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
out = model.generate(inputs, max_new_tokens=128)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

trust_remote_code=True is required — the architecture ships with the repository (modeling_mini_llm.py, configuration_mini_llm.py). Sampling defaults (temperature 0.5, top_k 40, repetition penalty 1.15) live in generation_config.json.

The training format is single-turn and fixed; chat_template.jinja implements it exactly:

User: <question>
Assistant: <answer>

Multi-turn history is never sent to the model: it anchors on its own previous answers and degrades when it sees them, so each question is answered independently.

Limitations

Small model trained on a modest budget: answers are often fluent and in the right format, but factual errors and hallucinations are common. Do not use it for factual lookup or any production task. English only.

Links

• Base model weights: https://huggingface.co/GautierP/MiniLLM-40M/tree/main/base
• Code, training pipeline and full documentation: https://github.com/Mini-llm-lab/Mini-LLM
• Evaluation protocol and scores: https://github.com/Mini-llm-lab/Mini-Eval