Title: Where to Begin: Efficient Pretraining via Sub-network Selection and Distillation
Primary area: infrastructure, software libraries, hardware, systems, etc.
Keywords: Small Language Models, Efficient Pretraining, Distillation, Pruning, Sub-network extraction

Abstract: Small language models (SLM) offer efficient, resource-friendly alternatives to large models while retaining strong task performance.
We propose a framework for efficiently pretraining SLMs that combines three synergistic ingredients.
First, we identify structurally sparse **sub-network initializations** that consistently outperform random and pruning-based baselines under fixed FLOP budgets.
Second, we employ **evolutionary search** to discover high-quality subnet initializations, yielding stronger pretrained starting points.
Third, we leverage established **knowledge distillation** from larger teachers to accelerate convergence and boost generalization. 
Our best model from evolutionary search, initialized with large language model (LLM) weights, achieves the same validation perplexity as a comparable Pythia SLM while requiring $9.2\times$ fewer pretraining tokens.
Our framework is fully reproducible and released publicly, providing a principled recipe for cost-efficient SLM development at scale.

TL;DR: We present a library for efficient pretraining of SLMs with sub-network extraction and distillation