WritingBench: A Comprehensive Benchmark for Generative Writing
Authors: name: Yuning Wu; name: Jiahao Mei; name: Ming Yan; name: Chenliang Li; name: SHaopeng Lai; name: Yuran Ren; name: Zijia Wang; name: Ji Zhang; name: Mengyue Wu; name: Qin Jin; name: Fei Huang
Published: 2025-03-07 08:56:20

Abstract: Recent advancements in large language models (LLMs) have significantly
enhanced text generation capabilities, yet evaluating their performance in
generative writing remains a challenge. Existing benchmarks primarily focus on
generic text generation or limited in writing tasks, failing to capture the
diverse requirements of high-quality written contents across various domains.
To bridge this gap, we present WritingBench, a comprehensive benchmark designed
to evaluate LLMs across 6 core writing domains and 100 subdomains, encompassing
creative, persuasive, informative, and technical writing. We further propose a
query-dependent evaluation framework that empowers LLMs to dynamically generate
instance-specific assessment criteria. This framework is complemented by a
fine-tuned critic model for criteria-aware scoring, enabling evaluations in
style, format and length. The framework's validity is further demonstrated by
its data curation capability, which enables 7B-parameter models to approach
state-of-the-art (SOTA) performance. We open-source the benchmark, along with
evaluation tools and modular framework components, to advance the development
of LLMs in writing.