Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with
  Architecture-Routed Mixture-of-Experts
Authors: name: Ganesh Jawahar, name: Haichuan Yang, name: Yunyang Xiong, name: Zechun Liu, name: Dilin Wang, name: Fei Sun, name: Meng Li, name: Aasish Pappu, name: Barlas Oguz, name: Muhammad Abdul-Mageed, name: Laks V. S. Lakshmanan, name: Raghuraman Krishnamoorthi, name: Vikas Chandra

Weight-sharing supernet has become a vital component for performance
estimation in the state-of-the-art (SOTA) neural architecture search (NAS)
frameworks. Although supernet can directly generate different subnetworks
without retraining, there is no guarantee for the quality of these subnetworks
because of weight sharing. In NLP tasks such as machine translation and
pre-trained language modeling, we observe that given the same model
architecture, there is a large performance gap between supernet and training
from scratch. Hence, supernet cannot be directly used and retraining is
necessary after finding the optimal architectures.
  In this work, we propose mixture-of-supernets, a generalized supernet
formulation where mixture-of-experts (MoE) is adopted to enhance the expressive
power of the supernet model, with negligible training overhead. In this way,
different subnetworks do not share the model weights directly, but through an
architecture-based routing mechanism. As a result, model weights of different
subnetworks are customized towards their specific architectures and the weight
generation is learned by gradient descent. Compared to existing weight-sharing
supernet for NLP, our method can minimize the retraining time, greatly
improving training efficiency. In addition, the proposed method achieves the
SOTA performance in NAS for building fast machine translation models, yielding
better latency-BLEU tradeoff compared to HAT, state-of-the-art NAS for MT. We
also achieve the SOTA performance in NAS for building memory-efficient
task-agnostic BERT models, outperforming NAS-BERT and AutoDistil in various
model sizes.