Fully Open Source Moxin-7B Technical Report
Authors: name: Pu Zhao; name: Xuan Shen; name: Zhenglun Kong; name: Yixin Shen; name: Sung-En Chang; name: Timothy Rupprecht; name: Lei Lu; name: Enfu Nan; name: Changdi Yang; name: Yumei He; name: Xingchen Xu; name: Yu Huang; name: Wei Wang; name: Yue Chen; name: Yong He; name: Yanzhi Wang
Published: 2024-12-08 02:01:46

Abstract: Recently, Large Language Models (LLMs) have undergone a significant
transformation, marked by a rapid rise in both their popularity and
capabilities. Leading this evolution are proprietary LLMs like GPT-4 and
GPT-o1, which have captured widespread attention in the AI community due to
their remarkable performance and versatility. Simultaneously, open-source LLMs,
such as LLaMA and Mistral, have made great contributions to the ever-increasing
popularity of LLMs due to the ease to customize and deploy the models across
diverse applications. Although open-source LLMs present unprecedented
opportunities for innovation and research, the commercialization of LLMs has
raised concerns about transparency, reproducibility, and safety. Many
open-source LLMs fail to meet fundamental transparency requirements by
withholding essential components like training code and data, and some use
restrictive licenses whilst claiming to be "open-source," which may hinder
further innovations on LLMs. To mitigate this issue, we introduce Moxin 7B, a
fully open-source LLM developed in accordance with the Model Openness Framework
(MOF), a ranked classification system that evaluates AI models based on model
completeness and openness, adhering to principles of open science, open source,
open data, and open access. Our model achieves the highest MOF classification
level of "open science" through the comprehensive release of pre-training code
and configurations, training and fine-tuning datasets, and intermediate and
final checkpoints. Experiments show that our model achieves superior
performance in zero-shot evaluation compared with popular 7B models and
performs competitively in few-shot evaluation.