Title: MoDE: Weight Denoising Towards Better LLM Performance through a Mixture of Domain Experts
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Mixture of Experts, Large Language Models

Abstract: Pruning in large language models (LLMs) is widely assumed to degrade performance, since most weights are considered essential contributors to model capacity; thus, existing methods primarily rely on training to retain accuracy. However, our empirical study challenges this view: we find that some weights behave as noise weights—despite appearing moderately important—whose removal can in fact improve in-domain accuracy. To this end, we first present the **DENoise** (Domain Expert weight deNoising) algorithm, which effectively removes domain-aware noise weights without requiring fine-tuning to achieve improvement; We further develop **MoDE** (Mixture of Domain Experts), which treats these in-domain optimal denoised models as experts and employs a bilevel trainable router to dynamically activate them, thereby enhancing out-of-domain generalization. Experimental results show that applying the **DENoise** algorithm yields 2–3% gains across benchmarks such as MMLU, MBPP, and GSM8K, while **MoDE** achieves an average improvement of over 1.1% against baseline models, all without introducing additional parameters or tuning overhead.

TL;DR: Pruning noise weights boosts in-domain accuracy; MoDE ensures cross-domain generalization.