Title: Learning Once, Routing Right: Information-Theoretic Gating for Online Continual Mixture-of-Experts
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: Continual Learning, Mixture-of-Experts, Minimum Excess Risk, Mixture-of-Experts Gating

Abstract: Continual Learning (CL) requires models to learn from sequential data streams while retaining previously acquired knowledge. While Mixture-of-Experts (MoE) models offer a promising solution through dynamic expert selection, a critical gap remains: their gating mechanisms lack principled design from a continual learning perspective. To bridge this gap, we investigate the impact of gating strategies on the achievable Minimum Excess Risk (MER) under the online CL setting, where data can be seen only once. Our key theoretical contribution reveals a novel connection between the MER and the mutual information between expert assignments and labels/outputs. Based on this foundation, we propose two innovative mutual information-based loss functions for both fully labeled and label-free settings. Furthermore, to ensure computational efficiency, we introduce a lightweight, matrix-based mutual information estimator with rigorous joint entropy formulation. Extensive experiments on MNIST, Fashion-MNIST, KMNIST, and EMNIST demonstrate our approach's superiority, achieving up to 12.3\% lower overall error and 3.9\% reduced forgetting with statistical significance.