# Neural Models for Measuring Confidence on Interactive Machine Translation Systems
## 4 Confidence Measures  
In this project, we studied the performance of four different CMs in the IPNMT field. Each MT field presents a new problem that must be overcome. In the case of IPNMT, the main problem is the computation time of the CMs. In an IPNMT environment, the translator is interactively working with the system to generate the final translations. We need to perform all the calculations in less than 100 ms [35], or the user could feel that the system does not respond instantly and break the workflow. There are several techniques that we can no longer apply due to their high computational time. For this reason, we used neural models that can perform the forward pass to obtain the confidence estimations inside this limitation.  
Researchers can apply CMs at different levels: document, sentence, and word. These CMs provide an estimation of correctness. If this value is higher than the threshold set, we classify the element as correct. As our main goal within the field of IPNMT field is to reduce the human effort of translating each sentence, we applied the CMs at the word level. The system classifies all the words from the translation as correct or incorrect, and the translator only has to check the words classified as incorrect. Principally, with this technique, we are reducing the number of words that the user has to type. At the same time, we reduce the cognitive cost that is expended reviewing the words.  
Figure 1 describes a conventional prefix-based IPNMT session with word-level CMs. At iteration 0, the system translates the source sentence (\(\hat{s}_{h}\)) and classifies all the words (\(CM\)). Then, at iteration 1, the user has to check the first word classified as incorrect, _\"is\"_.  
In this case, the word is correct, so the user does not perform any changes, and the word is validated. The system corrects translations from left to right, so previous words classified as correct are also validated and saved in the prefix (**p**). At iteration 2, the user checks the next word classified as incorrect, \"_a_\". The system has generated in the fourth position a word different from the reference. The user corrects the error, and the system generates a new translation. All the remaining words have been classified as correct, and the translation process finishes. As can be seen in the example, this interactive process no longer assures perfect translations. To achieve this goal, we need accurate CMs that never fail in the classification process.  
To train the CMs, we used four of the most common quality metrics in the MT field: Bleu, Meteor, Chr-F, and TER. These metrics can provide a quality score of the whole sentence using the ground truth translation as a reference. The main problem that we encounter is that to train our neural models, we need the quality scores from the words of the translations, and the metrics do not give this information directly. In order to obtain a quality score for each word of the translation, we use a strategy initially used to overcome the shortcoming of the sparsity of rewards, the reward shaping [36]. Reward shaping helps to distribute a reward between intermediate steps, which we use to distribute our sentence score between all the sentence words. In reward shaping, the intermediate reward for the word at position \(t\) of the translation \(y\) from the source sentence \(x\) is denoted as \(r_{t}(g_{t},x)\) and is calculated as follows:  
\[r_{t}(g_{t},x)=R(g_{1...t},x)-R(g_{1...t-1},x) \tag{1}\]  
where \(R(g_{1...t},x)\) is defined as the metric score of the translation \(g_{1...t}\) with respect to the source sentence \(x\). Note that to use reward shaping, the next equation has to be fulfilled:  
\[R(g,x)=\sum_{t=1}^{T}r_{t}(g_{1...t},x) \tag{2}\]  
where \(T\) is the total length of the translation.  
Each of the metrics that we selected in this project take into account different aspects when they calculate the translation scores. In the next section, we see more in detail the four metrics that we selected, and what we can expect from their CMs.