CascaMini-GPT2-210M-Edu-5B

CascaMini-GPT2-210M-Edu-5B is a small GPT-2-style causal language model trained from scratch as a learning project.

It is a base text-completion model, not an instruction-tuned assistant. It can generate educational-looking prose, but it may produce factual errors, circular explanations, repetition, invented details, and weak reasoning.

Model details

• Model family: CascaMini
• Model type: GPT-2-style decoder-only causal language model
• Parameters: approximately 209.5M
• Layers: 24
• Attention heads: 12
• Embedding size: 768
• Context length: 1024 tokens
• Vocabulary size: 50,257
• Tokenizer: GPT-2 BPE
• Training precision: CUDA training with mixed precision
• Training objective: next-token prediction
• Training tokens: approximately 5.24B tokens
• Best validation loss: 3.2245991230010986

Intended use

This model is intended for:

• educational experimentation
• text-completion experiments
• studying small language model training
• comparing GPT-2-style architecture against later CascaMini models

It is not intended for factual question answering, safety-critical use, professional advice, or production assistant use.

Training data

The model was trained on a local GPT-2-tokenised FineWeb-Edu dataset.

FineWeb-Edu is an educational-text subset filtered from FineWeb. The FineWeb-Edu dataset card describes it as containing 1.3T tokens, with an additional 5.4T-token score-2 version, and lists its licence as Open Data Commons Attribution License ODC-By v1.0, subject to Common Crawl terms.

Training procedure

This model was trained in two main phases:

1. Initial FineWeb-Edu pretraining to approximately 2.1B tokens.
2. Continuation training to approximately 5.24B total tokens.

The final released checkpoint is the best validation checkpoint from the continuation run.

Validation

Previous 2B checkpoint validation loss:

3.3519

Final 5B checkpoint validation loss:

3.2245991230010986

Approximate validation perplexity:

exp(3.2246) ≈ 25.1

Example usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "beardymcgee/cascamini-gpt2-210m-edu-5b"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.float16,
).to("cuda")

prompt = "The main purpose of an operating system is"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=160,
        do_sample=True,
        temperature=0.55,
        top_k=50,
        top_p=0.9,
        repetition_penalty=1.1,
        no_repeat_ngram_size=4,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Known limitations

This model is small and was trained as a learning project. It is not instruction tuned.

Known issues include:

• factual errors
• circular explanations
• repetitive lists
• topic drift
• weak arithmetic and reasoning
• invented technical details
• plausible but incorrect educational explanations

Development notes

CascaMini-GPT2-210M-Edu-5B is the first packaged model in the CascaMini series. The next planned model is a Llama-style under-400M parameter model using a more modern architecture.