CuriosLM/gpt2-124m-fineweb-edu
Model A: GPT-2 small (124M) trained from scratch, the baseline of a controlled comparison with a Gemma-style model trained identically.
A small research model from the CuriosLM project, released for reproducibility and study. It is a base model: not instruction-tuned and not safety-tuned, so it can produce false, biased or nonsensical text.
Read more in the write-up, Part 1, pretraining and Part 2, inference, positions, attention and context extension; the code is at Shinibali/CuriosLM.
Architecture
- GPT-2 small, unchanged: 12 layers, 12 heads, d_model 768, learned absolute positions (1,024), GELU MLP, pre-LayerNorm, biases on.
Training
- Data: 100% HuggingFaceFW/fineweb-edu (
sample-10BT) - Tokens: 9.83B, 20,000 optimizer steps
- AdamW, peak learning rate 0.0006 with 760 warmup steps and cosine decay to 6e-05, weight decay 0.1, seed 1337
- Tokens per step: 491,520
Results
| Evaluation | Score |
|---|---|
| Perplexity, held-out FineWeb-Edu | 21.89 |
| Perplexity, held-out Cosmopedia-v2 | 20.31 |
| ARC-Easy (acc) | 52.4% |
| HellaSwag (acc_norm) | 32.4% |
| LAMBADA (acc) | 24.2% |
| LAMBADA (perplexity) | 111.1 |
Perplexity on about 2M held-out tokens of each dataset; benchmarks zero-shot with EleutherAI's lm-evaluation-harness.
Usage
Standard 🤗 GPT-2 format:
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("CuriosLM/gpt2-124m-fineweb-edu")
model = AutoModelForCausalLM.from_pretrained("CuriosLM/gpt2-124m-fineweb-edu")
ids = tokenizer("The mitochondria is", return_tensors="pt").input_ids
print(tokenizer.decode(model.generate(ids, max_new_tokens=40, do_sample=True, top_k=50)[0]))
- Downloads last month
- 676