LLMagogyV4

“knowing a few principles frees you from knowing many facts” R. Descartes
“the knowledge of certain principles easily compensates the lack of knowledge of certain facts” C. Helvétius

Experimental research on methods of influencing LLM pre-training to achieve lower validation loss. This is the fourth iteration of the LLMagogy project series.

What is LLMagogy?

LLMagogy explores whether it is possible to influence the pre-training process of a language model through various interventions — staged training, regularization techniques, data ordering, and hyperparameter evolution — to achieve better generalization than standard training. Previous versions:
LLMagogy V1 — theoretical concept of multi-stage pre-training inspired by human learning https://github.com/loftyara/LLMagogy
LLMagogy V2 — experimental validation on a Python documentation dataset (model growth showed 5-25% time savings) https://github.com/loftyara/LLMagogyV2
LLMagogy V3 — failed attempt on TinyStories (results not published)

Key Results

Baseline (NanoGPT, 7.62M params, 3% TinyStories, BF16): val loss = 2.2026 after 5 epochs.
Effective methods found (table):

Method	Improvement	Notes
Stochastic Depth (before training)	~0.8%	prob=0.2, one epoch
Gradient Noise (after baseline)	~1.4%	std=0.00005, one epoch
Learning Rate reduction	~0.6%	LR to 0.0002
Global Warmup	minor	dropout=0.1, 25k iterations

Final synthesis (Stochastic Depth, then Gradient Noise, then LR reduction): val loss = 2.1270 (~3.5% improvement) in just ~2 epochs instead of 5.
Methods that did NOT work: precision evolution, model size growth, hyperparameter evolution (except LR), loss function modification, curriculum learning, warmup during model expansion.

Project Structure

Directory structure:

data/
   raw/ - Raw dataset files
   processed/ - Processed data
     random/ - Randomly split dataset
     curriculum/ - Dataset split by various criteria
docs/ - Documentation (PDF)
scripts/ - Data preparation scripts
    01_download_dataset.py
    02_prepare_text.py
    03_split_cumulative.py
    04_tokenize_bin.py
src/ - Training scripts
    01_train_baseline.py - Step 1: parameter selection
    02_train_baseline_detailed.py - Step 2: baseline training
    03_train_evolution_precision.py - Step 3: precision evolution
    04_train_evolution_structural.py - Step 4: size evolution
    05_train_evolution_hyperparams.py - Step 5: hyperparameter evolution
    06_train_evolution_loss.py - Step 6: loss function evolution
    07_train_evolution_regulation.py - Step 7: regularization methods
    08_train_evolution_datasets.py - Step 8: dataset evolution
    09_train_evolution_warmup.py - Step 9: size evolution with warmup
    10_train_global_warmup.py - Step 10: global warmup
    11_train_synthesis.py - Step 11: synthesis of effective methods

Requirements

Python 3.10+
PyTorch (with CUDA support)
RTX 5070 Ti (16 GB VRAM) or equivalent
All computations in bfloat16

Setup

Clone the repository:

git clone https://github.com/loftyara/LLMagogyV4.git

then

cd LLMagogyV4

Create virtual environment:

python -m venv venv

Activate virtual environment:

venv\Scripts\activate

(Windows), or

source venv/bin/activate

(Linux/Mac) Install dependencies:

pip install torch numpy datasets tiktoken

Dataset Preparation

cd scripts
python 01_download_dataset.py - Download TinyStories from Hugging Face
python 02_prepare_text.py - Split dataset into parts
python 03_split_cumulative.py - Split by curriculum criteria
python 04_tokenize_bin.py - Tokenize to .bin files

Training

All training scripts are in src/. Run them sequentially or individually.

cd src
python 01_train_baseline.py - Parameter selection (1 epoch per config)
python 02_train_baseline_detailed.py - Full baseline training (5 epochs)
...
python 11_train_synthesis.py - Final synthesis (best result)

Model Configuration

Parameter	Value
Architecture	NanoGPT (GPT-2 tokenizer, vocab_size=50257)
n_layer	6
n_embd	128
n_head	4
block_size	256
Parameters	7.62M
Precision	bfloat16
Optimizer	AdamW (betas=0.9/0.95, weight_decay=0.1, grad_clip=1.0)
batch_size	48
learning_rate	0.001
Training data	3% TinyStories
Validation data	50% TinyStories
eval_iters	500 fixed batches

Documentation

Full documentation with detailed experiment descriptions, results, and analysis: docs/llmagogyv4.pdf

CONTACT

Author: Dmitri Lyubimkov
Email: loftlong@gmail.com
GitHub: @loftyara