SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective
  Depth Up-Scaling
Authors: name: Dahyun Kim, name: Chanjun Park, name: Sanghoon Kim, name: Wonsung Lee, name: Wonho Song, name: Yunsu Kim, name: Hyeonwoo Kim, name: Yungi Kim, name: Hyeonju Lee, name: Jihoo Kim, name: Changbae Ahn, name: Seonghoon Yang, name: Sukyung Lee, name: Hyunbyung Park, name: Gyoungjin Gim, name: Mikyoung Cha, name: Hwalsuk Lee, name: Sunghun Kim

We introduce depth up-scaling (DUS), a novel technique to up-scale base LLMs
efficiently and effectively in a simple manner. In contrast to
mixture-of-experts (MoE), DUS does not require complex changes to train and
inference. Using DUS, we build SOLAR 10.7B, a large language model (LLM) with
10.7 billion parameters, demonstrating superior performance in various natural
language processing (NLP) tasks. Comparative evaluations show that SOLAR 10.7B
outperforms existing open-source pretrained LLMs, such as Llama 2 and Mistral
7B. We additionally present SOLAR 10.7B-Instruct, a variant fine-tuned for
instruction-following capabilities, surpassing Mixtral-8x7B. SOLAR 10.7B is
publicly available under the Apache 2.0 license, promoting broad access and
application in the LLM field.