SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective
  Depth Up-Scaling
Authors: name: Dahyun Kim; name: Chanjun Park; name: Sanghoon Kim; name: Wonsung Lee; name: Wonho Song; name: Yunsu Kim; name: Hyeonwoo Kim; name: Yungi Kim; name: Hyeonju Lee; name: Jihoo Kim; name: Changbae Ahn; name: Seonghoon Yang; name: Sukyung Lee; name: Hyunbyung Park; name: Gyoungjin Gim; name: Mikyoung Cha; name: Hwalsuk Lee; name: Sunghun Kim
Published: 2023-12-23 05:11:37

Abstract: We introduce depth up-scaling (DUS), a novel technique to up-scale base LLMs
efficiently and effectively in a simple manner. In contrast to
mixture-of-experts (MoE), DUS does not require complex changes to train and
inference. Using DUS, we build SOLAR 10.7B, a large language model (LLM) with
10.7 billion parameters, demonstrating superior performance in various natural
language processing (NLP) tasks. Comparative evaluations show that SOLAR 10.7B
outperforms existing open-source pretrained LLMs, such as Llama 2 and Mistral
7B. We additionally present SOLAR 10.7B-Instruct, a variant fine-tuned for
instruction-following capabilities, surpassing Mixtral-8x7B. SOLAR 10.7B is
publicly available under the Apache 2.0 license, promoting broad access and
application in the LLM field.