Title: GeneLLM: Inheriting 1.25% of MoE-LLMs to Build Models of 8% Size that Retain 80% Performance
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models

Abstract: Despite the rapid progress of large language models (LLMs), their enormous training costs and deployment challenges remain significant limitations. To address these issues, the Learngene framework was proposed, offering a one-time extraction of a compact, reusable knowledge module from a large model, which enables smaller models of various sizes to generalize more effectively across downstream tasks. In this work, we are the first to apply the Learngene framework to LLMs. We propose GeneLLM, a low-cost initialization framework that allows rapid construction of lightweight domain-specialized models under limited data resources. By performing only forward inference on Mixture-of-Experts (MoE) LLM with multi-domain data, we identify experts of LLM with cross-domain common knowledge and apply tensor decomposition to refine a compact knowledge module, termed learngene. This module is then used to reconstruct and initialize the Feed-Forward Network (FFN) layers of small dense models, enabling efficient adaptation to downstream tasks with minimal computational cost. Experiments show that by inheriting only 1.25% parameters of the source LLM, GeneLLM enables target lightweight models, only 8.1% the size of the source LLM, to recover over 80% of its performance on certain tasks. Compared to training from scratch and distillation, GeneLLM consistently delivers superior task performance, cost efficiency, and faster convergence speed. The code is available at https://anonymous.4open.science/r/GeneLLM-main-0EB3/.