Stronger Models are NOT Stronger Teachers for Instruction Tuning
Authors: name: Zhangchen Xu; name: Fengqing Jiang; name: Luyao Niu; name: Bill Yuchen Lin; name: Radha Poovendran
Published: 2024-11-11 17:06:48

Abstract: Instruction tuning has been widely adopted to ensure large language models
(LLMs) follow user instructions effectively. The resulting
instruction-following capabilities of LLMs heavily rely on the instruction
datasets used for tuning. Recently, synthetic instruction datasets have emerged
as an economically viable solution to provide LLMs diverse and high-quality
instructions. However, existing approaches typically assume that larger or
stronger models are stronger teachers for instruction tuning, and hence simply
adopt these models as response generators to the synthetic instructions. In
this paper, we challenge this commonly-adopted assumption. Our extensive
experiments across five base models and twenty response generators reveal that
larger and stronger models are not necessarily stronger teachers of smaller
models. We refer to this phenomenon as the Larger Models' Paradox. We observe
that existing metrics cannot precisely predict the effectiveness of response
generators since they ignore the compatibility between teachers and base models
being fine-tuned. We thus develop a novel metric, named as
Compatibility-Adjusted Reward (CAR) to measure the effectiveness of response
generators. Our experiments across five base models demonstrate that CAR
outperforms almost all baselines.