welmia-2.0-103m-base

A 103M-parameter GPT-style language model (RoPE, RMSNorm, SwiGLU, tied embeddings) pretrained completely from scratch on ~5B tokens (OpenWebText).

This is the base (non-instruction-tuned) model — raw next-token prediction only. For an instruction-following version, see welmia-1.0-103m-instruct (uploaded separately).

Architecture

Parameter	Value
Parameters	103.4M
Layers	12
Attention Heads	12
Embedding Dim	768
Context Length	512
Vocab Size	24,000
Normalization	RMSNorm
Activation	SwiGLU
Positional Enc.	RoPE
Weight Tying	Yes

Tokenizer note: this model uses a custom 24k-vocab BPE tokenizer trained specifically for this model — it is not the GPT-2/tiktoken vocabulary used by welmia-1.0-81m. Checkpoints and tokenizers between the two models are not interchangeable.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained("Welmia/welmia-2.0-103m-base", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("Welmia/welmia-2.0-103m-base")

ids = tok("The history of artificial intelligence began", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=100, do_sample=True, temperature=0.8, top_k=40)
print(tok.decode(out[0], skip_special_tokens=True))

trust_remote_code=True is required — this is a custom architecture, defined in modeling_gpt.py in this repo.

Important notes

• Base model, not instruction-tuned. It will continue text rather than answer questions or follow instructions. Use the -instruct version for chat/Q&A behavior.
• No KV cache in this implementation — generate() recomputes the full forward pass each step, so it's slower than production-grade inference code at longer output lengths.
• 512-token context. Inputs longer than this are truncated from the left during generation.

Training

Trained from scratch on ~5B tokens of OpenWebText, causal LM objective, RoPE + SwiGLU + RMSNorm architecture, AdamW optimizer with cosine LR schedule.

License

Apache License 2.0