Word confidence measures for machine translation

Due to continuous research which led to improved concepts and algorithms, the quality of automatically generated translation has significantly improved in recent years. However, the performance of machine translation systems is still not perfect. For human users dealing with these systems, it is desirable to obtain a reliable indication of possible errors in the system output. The same holds for applications based on machine translation technologies. They could explore the knowledge about possible mistakes. Interactive machine translation systems, for example, can modify or even discard predicted translations which are identified as possible errors. The aim of this work is to provide knowledge about when a translation generated by the system is incorrect by calculating measures of confidence for each word in this translation. This topic has hardly been investigated in machine translation before. Different ways of determining confidence measures are proposed and experimentally evaluated in this thesis. The basic concept behind all these approaches are word posterior probabilities. The goal is to set up a sound theoretical framework for the calculation and evaluation of confidence measures in machine translation. The main problem which has to be solved for the computation of word posterior probabilities is to define the underlying concept. There exists no intuitive definition of this concept. Possible approaches include the word posterior probability of a word based on its position in the sentence and the occurrence in any position. Several solutions to this problem are presented in this thesis. Furthermore, different approaches to the calculation of word posterior probabilities are introduced and compared. They can be divided into two categories: system-based methods which explore knowledge provided by the translation system that has generated the translations, and direct methods which are independent of the translation system. The system-based techniques make use of system output, such as word graphs or N -best lists. The word posterior probability is determined by summing the probabilities of all sentences which contain the target word. The direct confidence measures developed here take other knowledge sources, such as word or phrase lexica, into account. They can be applied to the output of non-statistical machine translation systems as well. The word posterior probabilities introduced in this thesis can directly be applied as confidence measures as follows: For a given translation generated by a machine translation system, the posterior probabilities of all words are determined and compared to a threshold. All words whose posterior probability is above this threshold are tagged as correct and all others are tagged as incorrect. To evaluate the proposed confidence measures, the information on which words are correct is needed. In machine translation, it is not intuitively clear how to determine the correctness of single words. As a solution to this problem, several different ways of deriving word error measures from existing machine translation evaluation metrics are suggested and investigated. The relation between the word error measure and the word posterior probabilities is studied in detail. From the formulation of the posterior risk for different error measures, a theoretical foundation of the word posterior probabilities is derived. The different confidence measures proposed here explore information from various knowledge sources, such as sentence probabilities provided by the machine translation system and statistical word and phrase lexica. To explore the knowledge from all these sources, a combination of several confidence measures is investigated. The suggested methods are evaluated on different translation tasks and several language pairs. In order to assess the general discriminative power of the confidence measures, they are tested on output from four different machine translation systems. Three of those are state-of-the-art phrase-based systems, and the fourth is an established rule-based system. A significant improvement in terms of confidence error rate is achieved in all settings. In this work, applications of confidence measures that improve translation quality of state-of-the-art systems are investigated. These include rescoring with confidence measures and their use in an interactive machine translation system. Rescoring with confidence measures is shown to improve translation quality. In the interactive machine translation environment, word confidence measures are successfully applied to select and discard possible translations based on their confidence. For the evaluation of the interactive translation experiments, an existing automatically determined metric is used. An extension to this metric is proposed to better model the gain achieved from using the system in a real-world application. The experiments show that the quality of the predicted translations is improved through the use of confidence measures.