LoftLiuping/llmagogyv2

📚 LLMagogy V2

Fast Multi-Stage Pre-Training of Large Language Models on Pre-Selected Limited Datasets

"Knowledge of a few principles frees one from knowledge of many facts." — R. Descartes

This repository contains the research report and experimental results for LLMagogy V2, an investigation into progressive model growth as an alternative to traditional end-to-end pre-training.

🔍 Key Findings

• ✅ Progressive growth on full dataset reduces training time by 5–25% without quality loss.
• ❌ Staged training on cumulative subsets (non-repetitive data) does not work — human-like curriculum fails on non-repetitive text.
• ✅ Random initialization of new weights > zero initialization.
• ✅ Warmup of new weights helps stabilize early training.
• ❌ Gradient freezing (weight_freeze < 1.0) provides no benefit.

📄 Full Report

📄 Read the full research paper:
👉 LLMagogyV2.pdf

💻 Code & Experiments

All code, datasets, and raw results are available on GitHub:
🔗 https://github.com/loftyara/LLMagogyV2

📊 Experimental Setup

• Dataset: Official Python documentation (structured, non-repetitive)
• Models: GPT-style transformers, grown from small to large
• Stages: Up to 5 stages of depth/width expansion
• Hardware: NVIDIA RTX 5070 Ti (16 GB VRAM)

📝 License

MIT License. See LICENSE.

➡️ What's Next?

LLMagogy V3 will explore:

• Datasets with repeating information (e.g., textbooks, tutorials),
• Optimal number of stages and growth strategies,
• Adaptive warmup and freezing policies.

Continuation follows...