QE4PE: Word-level Quality Estimation for Human Post-Editing
Authors: name: Gabriele Sarti, name: Vilém Zouhar, name: Grzegorz Chrupała, name: Ana Guerberof-Arenas, name: Malvina Nissim, name: Arianna Bisazza

Word-level quality estimation (QE) detects erroneous spans in machine
translations, which can direct and facilitate human post-editing. While the
accuracy of word-level QE systems has been assessed extensively, their
usability and downstream influence on the speed, quality and editing choices of
human post-editing remain understudied. Our QE4PE study investigates the impact
of word-level QE on machine translation (MT) post-editing in a realistic
setting involving 42 professional post-editors across two translation
directions. We compare four error-span highlight modalities, including
supervised and uncertainty-based word-level QE methods, for identifying
potential errors in the outputs of a state-of-the-art neural MT model.
Post-editing effort and productivity are estimated by behavioral logs, while
quality improvements are assessed by word- and segment-level human annotation.
We find that domain, language and editors' speed are critical factors in
determining highlights' effectiveness, with modest differences between
human-made and automated QE highlights underlining a gap between accuracy and
usability in professional workflows.