If You Can't Use Them, Recycle Them: Optimizing Merging at Scale
  Mitigates Performance Tradeoffs
Authors: name: Muhammad Khalifa, name: Yi-Chern Tan, name: Arash Ahmadian, name: Tom Hosking, name: Honglak Lee, name: Lu Wang, name: Ahmet Üstün, name: Tom Sherborne, name: Matthias Gallé

Model merging has shown great promise at combining expert models, but the
benefit of merging is unclear when merging ``generalist'' models trained on
many tasks. We explore merging in the context of large (sim100B) models, by
recycling checkpoints that exhibit tradeoffs among different tasks.
Such checkpoints are often created in the process of developing a frontier
model, and many suboptimal ones are usually discarded. Given a pool of model
checkpoints obtained from different training runs (e.g., different stages,
objectives, hyperparameters, and data mixtures), which naturally show tradeoffs
across different language capabilities (e.g., instruction following vs. code
generation), we investigate whether merging can recycle such suboptimal models
into a Pareto-optimal one. Our optimization algorithm tunes the weight of each
checkpoint in a linear combination, resulting in a Pareto-optimal models that
outperforms both individual models and merge-based baselines. Further analysis
shows that good merges tend to include almost all checkpoints with with
non-zero weights, indicating that even seemingly bad initial checkpoints can
contribute to good final merges.