LLMPretrain — a 101M decoder-only LM trained from scratch

Every weight in this model was produced by this project. There is no from_pretrained, no base checkpoint, and no adapter: the tokenizer, the architecture code, the data pipeline and the training loop are all bespoke, and training started from a random initialisation.

Trained on a single RTX 5070 Ti (16 GB).

Architecture

Written by hand — no transformers model classes were used to define it.

Type	decoder-only, pre-norm transformer
Parameters	100,684,032
Layers	12
d_model	768
Attention heads	12 query / 4 KV (GQA)
Head dim	64
FFN	SwiGLU, hidden 2048 (≈ 8/3 × d_model)
Normalisation	RMSNorm (pre-norm) + QK-norm
Positions	RoPE (θ=10000) — no learned position embeddings
Context	1024
Vocab	32,768 (own byte-level BPE, padded to a multiple of 128)
Biases	none, in any linear layer
Embeddings	tied between input and LM head

Parameter split: 25,165,824 embedding (25.0%) / 75,517,440 transformer blocks.

Training

Tokens seen	3,932,160,000
Steps	7,500
Schedule	warmup-stable-decay (WSD)
Optimizer	fused AdamW, betas (0.9, 0.95), wd 0.1 on 2D params only
Precision	bf16 autocast, fp32 master weights
Hardware	1 × RTX 5070 Ti 16 GB

Evaluation

Bits per byte is the headline metric. This model's vocabulary (32,768) differs from GPT-2's (50,257), and cross-entropy per token is not comparable across tokenizers. BPB divides that out and is directly comparable.

Metric	This model	GPT-2 124M
Parameters	100,684,032	124,439,808
Validation loss	3.1925	—
Bits per byte	1.0001	—
HellaSwag (0-shot, len-norm)	pending	29.55%

Validation loss is measured on a held-out slice of the training corpus and is not comparable to GPT-2's OpenWebText number. Bits per byte is, because it is tokenizer-independent.

Data provenance and licensing

Source	HF id	License
fineweb-edu	HuggingFaceFW/fineweb-edu	ODC-By 1.0

Attribution: FineWeb-Edu (HuggingFaceFW/fineweb-edu), ODC-By 1.0

ODC-By requires attribution, which is given above. Whether pretraining-data licenses flow through to model weights is legally unsettled; permissively licensed sources were chosen deliberately as the most defensible posture. Model weights are released under cc-by-nc-4.0.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Dikshan1234/LLMPretrain")
model = AutoModelForCausalLM.from_pretrained("Dikshan1234/LLMPretrain",
                                             trust_remote_code=True)

inputs = tok("Once upon a time", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=100, do_sample=True,
                     temperature=0.8, top_p=0.95)
print(tok.decode(out[0]))

trust_remote_code=True is required because the architecture (RoPE + SwiGLU + GQA + QK-norm) is not a stock transformers model. The two files it loads, configuration_llmpretrain.py and modeling_llmpretrain.py, are in this repo and are short enough to read.

Sample generations

===== step 7500 =====

--- [greedy] 'Once upon a time'
Once upon a time, the world was a place where people could live and work, and the world was a place where people could work and play.
The world was a place where people could work and play.
The world was a place where people could work and play.
The world was a place where people could work and play.
The world was a place where people could work and play.
The world was a place where people could work and play.
The world was a place where people could work and play.
The world was a place where people could work and play.
The world

--- [t=0.8] 'Once upon a time'
Once upon a time when there was no clear way of knowing that the country was going to survive another Ice Age, the Vikings tried to make out who was a country and who was a different.
This is not to say that they didn’t see how a few of them had a clue. But it is worth noting that there were many people who went to Scandinavia for the Vikings. One of them was named in the poem I have just written since he was around when he was a pupil of John of Gaunt. As he became famous, he became a monk and became known as a monk of St. John

--- [greedy] 'The capital of France is'
The capital of France is the city of Paris. The city is the capital of the French Republic.
The city of Paris is the capital of the French Republic. The city is the capital of the French Republic. The city is the capital of the French Republic. The city is the capital of the French Republic. The city is the capital of the French Republic. The city is the capital of the French Republic. The city is the capital of the French Republic. The city is the capital of the French Republic. The city is the capital of the French Republic. The city is the capital of the French Republic. The city

--- [t=0.8] 'The capital of France is'
The capital of France is located in the north of the country at the bottom of the river, bordered by the Swiss, the Netherlands, Luxembourg, France, and the Netherlands to the east.
The capital of the country is Paris, which is home to the largest collection of monuments.
The capital and largest city is Paris.
The city of Paris is located in the west of France, so it is located in the northwest of France.
The capital of the country is Paris.
The capital of the country is Paris.
The capital of the country is Paris.
The capital of the country is

--- [greedy] 'In 1969, humans first'
In 1969, humans first discovered the first known species of the genus Homo.
The genus Homo was named after the Greek word for “sister” or “sister.” The genus Homo was named after the Greek word for “sister.”
The genus Homo was named after the Greek word for “sister.”
The genus Homo was named after the Greek word for “sister.”
The genus Homo was named after the Greek word for “sister.”
The genus Homo was named after the Greek word for “sister.”
The genus Homo was named after the Greek word for “sister.”


--- [t=0.8] 'In 1969, humans first'
In 1969, humans first discovered the brain of a chimpanzee, named Benny, who has been described as the ‘first chimpanzee to walk on a toilet’ (Heller). The ‘first chimpanzee’ became the ‘first human to walk on a toilet’ and became ‘the first chimpanzee to walk on a toilet’. Since then, chimpanzees have been used in experiments like the experiments of Hans and T. H. Bieber, the first people to walk on a toilet.
There are many differences between chimpanzees and other primates. Chimpanzees (pimp) have the same

--- [greedy] 'The three states of matter are'
The three states of matter are the same, but the two states of matter are different.
The two states of matter are the same, but the two states of matter are different.
The two states of matter are the same, but the two states of matter are different.
The two states of matter are different, but the two states of matter are different.
The two states of matter are different, but the two states of matter are different.
The two states of matter are different, but the two state

Limitations

This is a base model trained on a modest token budget. It has had no instruction tuning, no RLHF and no safety alignment. It will produce confidently wrong text, and it reflects whatever biases exist in its training corpus. It is a research and educational artifact, not a product.