The Devil is in the Errors: Leveraging Large Language Models for
  Fine-grained Machine Translation Evaluation
Authors: name: Patrick Fernandes; name: Daniel Deutsch; name: Mara Finkelstein; name: Parker Riley; name: André F. T. Martins; name: Graham Neubig; name: Ankush Garg; name: Jonathan H. Clark; name: Markus Freitag; name: Orhan Firat
Published: 2023-08-14 17:17:21

Abstract: Automatic evaluation of machine translation (MT) is a critical tool driving
the rapid iterative development of MT systems. While considerable progress has
been made on estimating a single scalar quality score, current metrics lack the
informativeness of more detailed schemes that annotate individual errors, such
as Multidimensional Quality Metrics (MQM). In this paper, we help fill this gap
by proposing AutoMQM, a prompting technique which leverages the reasoning and
in-context learning capabilities of large language models (LLMs) and asks them
to identify and categorize errors in translations. We start by evaluating
recent LLMs, such as PaLM and PaLM-2, through simple score prediction
prompting, and we study the impact of labeled data through in-context learning
and finetuning. We then evaluate AutoMQM with PaLM-2 models, and we find that
it improves performance compared to just prompting for scores (with
particularly large gains for larger models) while providing interpretability
through error spans that align with human annotations.