BAM! Just Like That: Simple and Efficient Parameter Upcycling for
  Mixture of Experts
Authors: name: Qizhen Zhang, name: Nikolas Gritsch, name: Dwaraknath Gnaneshwar, name: Simon Guo, name: David Cairuz, name: Bharat Venkitesh, name: Jakob Foerster, name: Phil Blunsom, name: Sebastian Ruder, name: Ahmet Ustun, name: Acyr Locatelli

The Mixture of Experts (MoE) framework has become a popular architecture for
large language models due to its superior performance over dense models.
However, training MoEs from scratch in a large-scale regime is prohibitively
expensive. Existing methods mitigate this by pre-training multiple dense expert
models independently and using them to initialize an MoE. This is done by using
experts' feed-forward network (FFN) to initialize the MoE's experts while
merging other parameters. However, this method limits the reuse of dense model
parameters to only the FFN layers, thereby constraining the advantages when
"upcycling" these models into MoEs. We propose BAM (Branch-Attend-Mix), a
simple yet effective method that addresses this shortcoming. BAM makes full use
of specialized dense models by not only using their FFN to initialize the MoE
layers but also leveraging experts' attention parameters fully by initializing
them into a soft-variant of Mixture of Attention (MoA) layers. We explore two
methods for upcycling attention parameters: 1) initializing separate attention
experts from dense models including all attention parameters for the best model
performance; and 2) sharing key and value parameters across all experts to
facilitate for better inference efficiency. To further improve efficiency, we
adopt a parallel attention transformer architecture to MoEs, which allows the
attention experts and FFN experts to be computed concurrently. Our experiments
on seed models ranging from 590 million to 2 billion parameters demonstrate
that BAM surpasses baselines in both perplexity and downstream task
performance, within the same computational and data constraints.