Title: ASMG: Data Structure-Aware Routing via Incremental Subspace Learning for MoE
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Mixture of Experts, Input-Aware Routing, MoE, GHA, Unsupervised Learning

Abstract: Mixture-of-Experts (MoE) models scale model capacity efficiently by selectively
routing inputs to a subset of specialized experts. However, their performance
critically hinges on the gating mechanism, which is typically implemented as a
shallow linear projection followed by a softmax or sigmoid activation. This minimal
design lacks the representational capacity to capture structural variations in the
input, leading to poor expert specialization and suboptimal routing. To address this,
we propose Adaptive Structure-Aware MoE Gating (ASMG), a data-driven gating
mechanism that dynamically interpolates between a standard learnable gating
matrix and an evolving principal subspace learned via the Generalized Hebbian
Algorithm (GHA). By tracking input structure with iterative basis update with GHA,
ASMG enables the gating function to remain both learnable and structure-aware
throughout training. We validate our method through two controlled synthetic tasks:
(i) Gaussian mixture classification and (ii) sequence modeling from multinomial
HMMs, demonstrating improved expert-input alignment and routing specialization.
Extending to language and vision benchmarks, ASMG achieves consistent gains
over MoE baselines. Moreover, optionally enabling unsupervised GHA updates at
test time further improves robustness under distribution shifts, offering an online
adaptation mechanism that upgrades standard gating with stronger OOD resilience.

TL;DR: We propose ASMG, a structure-aware gating mechanism for Mixture-of-Experts that combines learnable gating with an incremental principal subspace learning, improving expert specialization and routing across synthetic, language, and vision tasks.