DITING: A Multi-Agent Evaluation Framework for Benchmarking Web Novel
  Translation
Authors: name: Enze Zhang; name: Jiaying Wang; name: Mengxi Xiao; name: Jifei Liu; name: Ziyan Kuang; name: Rui Dong; name: Eric Dong; name: Sophia Ananiadou; name: Min Peng; name: Qianqian Xie
Published: 2025-10-10 08:10:10

Abstract: Large language models (LLMs) have substantially advanced machine translation
(MT), yet their effectiveness in translating web novels remains unclear.
Existing benchmarks rely on surface-level metrics that fail to capture the
distinctive traits of this genre. To address these gaps, we introduce DITING,
the first comprehensive evaluation framework for web novel translation,
assessing narrative and cultural fidelity across six dimensions: idiom
translation, lexical ambiguity, terminology localization, tense consistency,
zero-pronoun resolution, and cultural safety, supported by over 18K
expert-annotated Chinese-English sentence pairs. We further propose AgentEval,
a reasoning-driven multi-agent evaluation framework that simulates expert
deliberation to assess translation quality beyond lexical overlap, achieving
the highest correlation with human judgments among seven tested automatic
metrics. To enable metric comparison, we develop MetricAlign, a meta-evaluation
dataset of 300 sentence pairs annotated with error labels and scalar quality
scores. Comprehensive evaluation of fourteen open, closed, and commercial
models reveals that Chinese-trained LLMs surpass larger foreign counterparts,
and that DeepSeek-V3 delivers the most faithful and stylistically coherent
translations. Our work establishes a new paradigm for exploring LLM-based web
novel translation and provides public resources to advance future research.