DocReward: A Document Reward Model for Structuring and Stylizing
Authors: name: Junpeng Liu, name: Yuzhong Zhao, name: Bowen Cao, name: Jiayu Ding, name: Yilin Jia, name: Tengchao Lv, name: Yupan Huang, name: Shaohan Huang, name: Nan Yang, name: Li Dong, name: Lei Cui, name: Tao Ge, name: Xun Wang, name: Huitian Jiao, name: Sun Mao, name: FNU Kartik, name: Si-Qing Chen, name: Wai Lam, name: Furu Wei
Organization: Microsoft Research

Recent advances in agentic workflows have enabled the automation of tasks
such as professional document generation. However, they primarily focus on
textual quality, neglecting visual structure and style, which are crucial for
readability and engagement. This gap arises mainly from the absence of suitable
reward models to guide agentic workflows toward producing documents with
stronger structural and stylistic quality. To address this, we propose
DocReward, a document reward model that evaluates documents based on their
structure and style. We construct a multi-domain dataset DocPair of 117K paired
documents, covering 32 domains and 267 document types, each including a high-
and low-professionalism document with identical content but different structure
and style. This enables the model to evaluate professionalism comprehensively,
and in a textual-quality-agnostic way. DocReward is trained using the
Bradley-Terry loss to score documents, penalizing predictions that contradict
the annotated ranking. To assess the performance of reward models, we create a
test dataset containing document bundles ranked by well-educated human
evaluators. Notably, DocReward outperforms GPT-4o and GPT-5 in accuracy by 30.6
and 19.4 percentage points, respectively, demonstrating its superiority over
baselines. In an extrinsic evaluation of document generation, DocReward
achieves a significantly higher win rate of 60.8%, compared to GPT-5's 37.7%
win rate, demonstrating its utility in guiding generation agents toward
producing human-preferred documents.