Unsupervised Word-level Quality Estimation for Machine Translation
  Through the Lens of Annotators (Dis)agreement
Authors: name: Gabriele Sarti, name: Vilém Zouhar, name: Malvina Nissim, name: Arianna Bisazza

Word-level quality estimation (WQE) aims to automatically identify
fine-grained error spans in machine-translated outputs and has found many uses,
including assisting translators during post-editing. Modern WQE techniques are
often expensive, involving prompting of large language models or ad-hoc
training on large amounts of human-labeled data. In this work, we investigate
efficient alternatives exploiting recent advances in language model
interpretability and uncertainty quantification to identify translation errors
from the inner workings of translation models. In our evaluation spanning 14
metrics across 12 translation directions, we quantify the impact of human label
variation on metric performance by using multiple sets of human labels. Our
results highlight the untapped potential of unsupervised metrics, the
shortcomings of supervised methods when faced with label uncertainty, and the
brittleness of single-annotator evaluation practices.