Title: Low Rank Experts Enable Specialization In Dense Transformers
Primary area: foundation or frontier models, including LLMs
Keywords: transformers, moe, llms

Abstract: We present Low Rank Experts (LoREs), a drop‑in augmentation for standard Transformer MLPs that improves model performance. Each MLP hosts a router that selects a token‑specific top‑$k$ subset from a bank of low‑rank matrices. Their combined contribution is injected into the up‑projection, yielding a dynamic, per‑token rank-$k$ update to the base weight and executed in parallel with the up-projection via grouped GEMMs. To compare with dense baselines, we match the parameter budget of LoRE augmented transformers by shrinking the base expansion factor to offset the router and low-rank experts' parameters. Overall FLOPS decrease because the low‑rank branch is sparsely activated. LoREs delivers token-level specialization by routing each token to structured experts inside a single dense MLP. We observe consistent quality improvements on benchmark tasks for models with upto 1.6B parameters trained on 400B tokens.

TL;DR: We present routed low rank experts, a drop-in augmentation for standard Transformer MLPs that improves model performance.