Weaver: Foundation Models for Creative Writing
Authors: name: Tiannan Wang; name: Jiamin Chen; name: Qingrui Jia; name: Shuai Wang; name: Ruoyu Fang; name: Huilin Wang; name: Zhaowei Gao; name: Chunzhao Xie; name: Chuou Xu; name: Jihong Dai; name: Yibin Liu; name: Jialong Wu; name: Shengwei Ding; name: Long Li; name: Zhiwei Huang; name: Xinle Deng; name: Teng Yu; name: Gangan Ma; name: Han Xiao; name: Zixin Chen; name: Danjun Xiang; name: Yunxia Wang; name: Yuanyuan Zhu; name: Yi Xiao; name: Jing Wang; name: Yiru Wang; name: Siran Ding; name: Jiayang Huang; name: Jiayi Xu; name: Yilihamu Tayier; name: Zhenyu Hu; name: Yuan Gao; name: Chengfeng Zheng; name: Yueshu Ye; name: Yihang Li; name: Lei Wan; name: Xinyue Jiang; name: Yujie Wang; name: Siyu Cheng; name: Zhule Song; name: Xiangru Tang; name: Xiaohua Xu; name: Ningyu Zhang; name: Huajun Chen; name: Yuchen Eleanor Jiang; name: Wangchunshu Zhou
Published: 2024-01-30 18:58:43

Abstract: This work introduces Weaver, our first family of large language models (LLMs)
dedicated to content creation. Weaver is pre-trained on a carefully selected
corpus that focuses on improving the writing capabilities of large language
models. We then fine-tune Weaver for creative and professional writing purposes
and align it to the preference of professional writers using a suit of novel
methods for instruction data synthesis and LLM alignment, making it able to
produce more human-like texts and follow more diverse instructions for content
creation. The Weaver family consists of models of Weaver Mini (1.8B), Weaver
Base (6B), Weaver Pro (14B), and Weaver Ultra (34B) sizes, suitable for
different applications and can be dynamically dispatched by a routing agent
according to query complexity to balance response quality and computation cost.
Evaluation on a carefully curated benchmark for assessing the writing
capabilities of LLMs shows Weaver models of all sizes outperform generalist
LLMs several times larger than them. Notably, our most-capable Weaver Ultra
model surpasses GPT-4, a state-of-the-art generalist LLM, on various writing
scenarios, demonstrating the advantage of training specialized LLMs for writing
purposes. Moreover, Weaver natively supports retrieval-augmented generation
(RAG) and function calling (tool usage). We present various use cases of these
abilities for improving AI-assisted writing systems, including integration of
external knowledge bases, tools, or APIs, and providing personalized writing
assistance. Furthermore, we discuss and summarize a guideline and best
practices for pre-training and fine-tuning domain-specific LLMs.