Don't Throw Away Your Pretrained Model
Authors: name: Shangbin Feng, name: Wenhao Yu, name: Yike Wang, name: Hongming Zhang, name: Yulia Tsvetkov, name: Dong Yu
Organization: Tencent

Alignment training has tradeoffs: it helps language models (LMs) gain in
reasoning and instruction following but might lose out on skills such as
creativity and calibration, where unaligned base models are better at. We aim
to make the best of both worlds through model collaboration, where different
models in the training pipeline collaborate and complement each other. Since LM
responses feature interleaving skills that favor different models, we propose
Switch Generation, where pretrained and aligned model versions take turns to
``speak'' in a response sequence. Specifically, we train a switcher LM by
learning from outcomes of choosing different models to generate the next
segment across diverse queries and contexts. At inference time, the switcher LM
guides different model checkpoints to dynamically generate the next segment
where their strengths are most needed. Extensive experiments with 8 model
collaboration baselines and 18 datasets show that 1) model collaboration
consistently outperforms individual models on 16 out of 18 tasks, and 2) Switch
Generation further outperforms baselines by 12.9% on average. Further analysis
reveals that Switch Generation discovers compositional skills to solve problems
where individual models struggle and generalizes to unseen models and tasks,
reusing and repurposing by-products in expensive model training pipelines that
are otherwise discarded.