SmolTulu: Higher Learning Rate to Batch Size Ratios Can Lead to Better
  Reasoning in SLMs
Authors: name: Sultan Alrashed
Published: 2024-12-11 12:41:36

Abstract: We present SmolTulu-1.7b-Instruct, referenced in this report as
SmolTulu-DPO-1130, an instruction-tuned language model that adapts AllenAI's
Tulu 3 post-training pipeline to enhance Huggingface's SmolLM2-1.7B base model.
Through comprehensive empirical analysis using a 135M parameter model, we
demonstrate that the relationship between learning rate and batch size
significantly impacts model performance in a task-dependent manner. Our
findings reveal a clear split: reasoning tasks like ARC and GSM8K benefit from
higher learning rate to batch size ratios, while pattern recognition tasks such
as HellaSwag and IFEval show optimal performance with lower ratios. These
insights informed the development of SmolTulu, which achieves state-of-the-art
performance among sub-2B parameter models on instruction following, scoring
67.7% on IFEval (Delta11%), and mathematical reasoning with 51.6% on GSM8K
(Delta3.4%), with an alternate version achieving scoring 57.1% on ARC
(Delta5.4%). We release our model, training recipes, and ablation studies to
facilitate further research in efficient model alignment, demonstrating that
careful adaptation of optimization dynamics can help bridge the capability gap
between small and large language models.