welmia-2.0-103m-instruct

A 103M-parameter GPT-style language model (RoPE, RMSNorm, SwiGLU, tied embeddings), pretrained from scratch on ~5B tokens (OpenWebText), then instruction-tuned on the Alpaca dataset.

Base (non-instruct) version: Welmia/welmia-2.0-103m-base.

Architecture

Parameter	Value
Parameters	103.4M
Layers	12
Attention Heads	12
Embedding Dim	768
Context Length	512
Vocab Size	24,000
Normalization	RMSNorm
Activation	SwiGLU
Positional Enc.	RoPE
Weight Tying	Yes

Tokenizer note: custom 24k-vocab BPE tokenizer, not GPT-2/tiktoken. Not interchangeable with welmia-1.0-81m.

Prompt format

This model expects the Alpaca instruction format:

Instruction:

{your instruction}

Response:

or, with additional context:

Instruction:

{your instruction}

Input:

{context}

Response:

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained("Welmia/welmia-2.0-103m-instruct", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("Welmia/welmia-2.0-103m-instruct")

prompt = "### Instruction:\nExplain what the sun is in one sentence.\n\n### Response:\n"
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=150, do_sample=True, temperature=0.7, top_k=40)
print(tok.decode(out[0], skip_special_tokens=True))

trust_remote_code=True is required — custom architecture defined in modeling_gpt.py in this repo.

Important notes

• Single-turn only. Trained on Alpaca, which has no multi-turn/chat structure — treat each prompt as independent, not a running conversation.
• No KV cache. generate() recomputes the full forward pass each step.
• 512-token context. Longer inputs are truncated from the left.
• Small model — expect basic instruction-following, not strong reasoning or world knowledge.

Training

Pretrained from scratch on 5B tokens OpenWebText (causal LM), then fine-tuned on the Alpaca instruction dataset (52k examples, 3 epochs, AdamW, cosine LR, loss masked over the prompt so only the response is trained on).

License

Apache License 2.0