TeacherLM: Teaching to Fish Rather Than Giving the Fish, Language
  Modeling Likewise
Authors: name: Nan He; name: Hanyu Lai; name: Chenyang Zhao; name: Zirui Cheng; name: Junting Pan; name: Ruoyu Qin; name: Ruofan Lu; name: Rui Lu; name: Yunchen Zhang; name: Gangming Zhao; name: Zhaohui Hou; name: Zhiyuan Huang; name: Shaoqing Lu; name: Ding Liang; name: Mingjie Zhan
Published: 2023-10-29 14:16:54

Abstract: Large Language Models (LLMs) exhibit impressive reasoning and data
augmentation capabilities in various NLP tasks. However, what about small
models? In this work, we propose TeacherLM-7.1B, capable of annotating relevant
fundamentals, chain of thought, and common mistakes for most NLP samples, which
makes annotation more than just an answer, thus allowing other models to learn
"why" instead of just "what". The TeacherLM-7.1B model achieved a zero-shot
score of 52.3 on MMLU, surpassing most models with over 100B parameters. Even
more remarkable is its data augmentation ability. Based on TeacherLM-7.1B, we
augmented 58 NLP datasets and taught various student models with different
parameters from OPT and BLOOM series in a multi-task setting. The experimental
results indicate that the data augmentation provided by TeacherLM has brought
significant benefits. We will release the TeacherLM series of models and
augmented datasets as open-source.