Configurable Foundation Models: Building LLMs from a Modular Perspective
Authors: name: Chaojun Xiao, name: Zhengyan Zhang, name: Chenyang Song, name: Dazhi Jiang, name: Feng Yao, name: Xu Han, name: Xiaozhi Wang, name: Shuo Wang, name: Yufei Huang, name: Guanyu Lin, name: Yingfa Chen, name: Weilin Zhao, name: Yuge Tu, name: Zexuan Zhong, name: Ao Zhang, name: Chenglei Si, name: Khai Hao Moo, name: Chenyang Zhao, name: Huimin Chen, name: Yankai Lin, name: Zhiyuan Liu, name: Jingbo Shang, name: Maosong Sun
Organization: OpenBMB

Advancements in LLMs have recently unveiled challenges tied to computational
efficiency and continual scalability due to their requirements of huge
parameters, making the applications and evolution of these models on devices
with limited computation resources and scenarios requiring various abilities
increasingly cumbersome. Inspired by modularity within the human brain, there
is a growing tendency to decompose LLMs into numerous functional modules,
allowing for inference with part of modules and dynamic assembly of modules to
tackle complex tasks, such as mixture-of-experts. To highlight the inherent
efficiency and composability of the modular approach, we coin the term brick to
represent each functional module, designating the modularized structure as
configurable foundation models. In this paper, we offer a comprehensive
overview and investigation of the construction, utilization, and limitation of
configurable foundation models. We first formalize modules into emergent bricks
- functional neuron partitions that emerge during the pre-training phase, and
customized bricks - bricks constructed via additional post-training to improve
the capabilities and knowledge of LLMs. Based on diverse functional bricks, we
further present four brick-oriented operations: retrieval and routing, merging,
updating, and growing. These operations allow for dynamic configuration of LLMs
based on instructions to handle complex tasks. To verify our perspective, we
conduct an empirical analysis on widely-used LLMs. We find that the FFN layers
follow modular patterns with functional specialization of neurons and
functional neuron partitions. Finally, we highlight several open issues and
directions for future research. Overall, this paper aims to offer a fresh
modular perspective on existing LLM research and inspire the future creation of
more efficient and scalable foundational models.