SlowGuess/ACL_2020_OCR huggingface dataset:
Jierui Li $^{1}$  Lemao Liu $^{2*}$  Huayang Li $^{2}$  Guanlin Li $^{3}$  Guoping Huang $^{2}$  Shuming Shi $^{2}$
$^{2}$ Tencent AI Lab,  $^{3}$ Harbin Institute of Technology
{lijierui19,epsilonlee.green}@gmail.com,
{redmondliu,alanili,donkeyhuang,shumingshi}@tencent.com
Recently many efforts have been devoted to interpreting the black-box NMT models, but little progress has been made on metrics to evaluate explanation methods. Word Alignment Error Rate can be used as such a metric that matches human understanding, however, it can not measure explanation methods on those target words that are not aligned to any source word. This paper thereby makes an initial attempt to evaluate explanation methods from an alternative viewpoint. To this end, it proposes a principled metric based on fidelity in regard to the predictive behavior of the NMT model. As the exact computation for this metric is intractable, we employ an efficient approach as its approximation. On six standard translation tasks, we quantitatively evaluate several explanation methods in terms of the proposed metric and we reveal some valuable findings for these explanation methods in our experiments.
Neural machine translation (NMT) has witnessed great success during recent years (Sutskever et al., 2014; Bahdanau et al., 2014; Gehring et al., 2017; Vaswani et al., 2017). One of the main reasons is that neural networks possess the powerful ability to model sufficient context by entangling all source words and target words from translation history. The downside yet is its poor interpretability: it is unclear which specific words from the entangled context are crucial for NMT to make a translation decision. As interpretability is important for understanding and debugging the translation process and particularly to further improve NMT models, many efforts have been devoted to explanation methods for NMT (Ding et al., 2017; Alvarez-Melis and Jaakkola, 2017; Li et al., 2019;
Ding et al., 2019; He et al., 2019). However, little progress has been made on evaluation metric to study how good these explanation methods are and which method is better than others for NMT.
Generally speaking, we recognize two orthogonal dimensions for evaluating the explanation methods: i) how much the pattern (such as source words) extracted by an explanation method matches human understanding on predicting a target word; or ii) how the pattern matches predictive behavior of the NMT model on predicting a target word. In terms of i), Word Alignment Error Rate (AER) can be used as a metric to evaluate an explanation method by measuring agreement between human-annotated word alignment and that derived from the explanation method. However, AER can not measure explanation methods on those target words that are not aligned to any source words according to human annotation.
In this paper, we thereby make an initial attempt to measure explanation methods for NMT according to the second dimension of interpretability, which covers all target words. The key to our approach can be highlighted as fidelity: when extracting the most relevant words with an explanation method, if those relevant words have the potential to construct an optimal proxy model that agrees well with the NMT model on making a translation decision, then this explanation method is good (§3). To this end, we formalize a principled evaluation metric as an optimization problem over the expected disagreement between the optimal proxy model and the NMT model(§3.1). Since it is intractable to exactly calculate the principled metric for a given explanation method, we propose an approximate metric to address the optimization problem. Specifically, inspired by statistical learning theory (Vapnik, 1999), we cast the optimization problem into a standard machine learning problem which is addressed in a two-step strat
*This work was done during J.Li & G.Li's internship at Tencent AI Lab. L.Liu is the corresponding author.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 365-375 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
egy: firstly we follow empirical risk minimization to optimize the empirical risk; then we validate the optimized parameters on a held-out test dataset. Moreover, we construct different proxy model architectures by utilizing the most relevant words to make a translation decision, leading to variant approximate metric in implementation (§3.2).
We apply the approximate metric to evaluate four explanation methods including attention (Bahdanau et al., 2014; Vaswani et al., 2017), gradient norm (Li et al., 2016), weighted gradient (Ding et al., 2019) and prediction difference (Li et al., 2019). We conduct extensive experiments on three standard translation tasks for two popular translation models in terms of the proposed evaluation metric. Our experiments reveal valuable findings for these explanation methods: 1) The evaluation methods (gradient norm and prediction difference) are good to interpret the behavior of NMT; 2) The prediction difference performs better than other methods.
- It presents an attempt at evaluating the explanation methods for neural machine translation from a new viewpoint of fidelity.
- It proposes a principled metric for evaluation, and to put it into practice it derives a simple yet efficient approach to approximately calculate the metric.
- It quantitatively compares several different explanation methods and evaluates their effects in terms of the proposed metric.
2 NMT and Explanation Methods
2.1 NMT Models
Suppose  $\pmb{x} = \{x_1, \dots, x_{|\pmb{x}|}\}$  denotes a source sentence with length  $|\pmb{x}|$  and  $\pmb{y} = \{y_1, \dots, y_{|\pmb{y}|}\}$  is a target sentence. Most NMT literature models the following conditional probability  $P(\pmb{y} \mid \pmb{x})$  in an encoder-decoder fashion:
$$
\begin{array}{r c l} P (\boldsymbol {y} \mid \boldsymbol {x}) & = & \prod_ {t} P (y _ {t} \mid \boldsymbol {y} _ {<   t}, \boldsymbol {x}) \\ & = & \prod_ {t} P (y _ {t} \mid s _ {t}), \end{array} \tag {1}
$$
where  $\mathbf{y}_{<t} = \{y_1, \dots, y_{t-1}\}$  denotes a prefix of  $\mathbf{y}$  with length  $t - 1$ , and  $s_t$  is the decoding state vector of timestep  $t$ . In the encoding stage, the encoder of a NMT model transforms the source sentence  $\mathbf{x}$  into a sequence of hidden vectors  $h =$
$\{h_1,\dots ,h_{|\pmb{x}|}\}$  . In the decoding stage, the decoder module summarizes the hidden vectors  $\pmb{h}$  and the history decoding states  $s_{< t} = \{s_1,\dots ,s_{t - 1}\}$  into the decoding state vector  $s_t$  . In this paper, we consider two popular NMT translation architectures, RNN-SEARCH (Bahdanau et al., 2014) and TRANSFORMER (Vaswani et al., 2017). RNNSEARCH utilizes a bidirectional RNN to define  $\pmb{h}$  and it computes  $s_t$  by the attention function over  $\pmb{h}$  , i.e.,
$$
s _ {t} = \operatorname {A t t n} \left(s _ {t - 1}, \boldsymbol {h}\right), \tag {2}
$$
where Attn is the attention function, which is defined as follows:
$$
\begin{array}{l} \operatorname {A t t n} (q, \boldsymbol {v}) = \sum_ {i} \alpha (q, v _ {i}) v _ {i}, \\ \alpha (q, v _ {i}) = \frac {\exp (e (q , v _ {i}))}{\sum_ {j} \exp (e (q , v _ {j}))}, \tag {3} \\ \end{array}
$$
where  $q$  and  $v_{i}$  are vectors,  $e$  is a similarity function over a pair of vectors and  $\alpha$  is its normalized function.
Different from RNN-SEARCH, which relies on RNN, TRANSFORMER employs an attention network to define  $\pmb{h}$ , and two additional attention networks to define  $s_t$  as follows:
$$
\begin{array}{l} s _ {t} = \mathrm {A t t n} (s _ {t + \frac {1}{2}}, \pmb {h}), \\ s _ {t + \frac {1}{2}} = \mathrm {A t t n} (s _ {t - 1}, s _ {<   t}). \\ \end{array}
$$
In this section, we describe several popular explanation methods that will be evaluated with our proposed metric. Suppose  $c_{t} = \langle \pmb{y}_{< t},\pmb{x}\rangle$  denotes the context at timestep  $t$ ,  $w$  (or  $w^{\prime}$ ) denotes either a source or a target word in the context  $c_{t}$ . According to Poerner et al. (2018), each explanation method for NMT could be regarded as a word relevance score function  $\phi (w;y,c_t)$ , where  $\phi (w;y,c_t) > \phi (w';y,c_t)$  indicates that  $w$  is more useful for the translation decision  $P(y_{t}|c_{t})$  than word  $w^{\prime}$ .
Attention Since Bahdanau et al. (2014) propose the attention mechanism for NMT, it has been the most popular explanation method for NMT (Tu et al., 2016; Mi et al., 2016; Liu et al., 2016; Zenkel et al., 2019).
<sup>1</sup>Due to space limitation, we present the notations for a single layer NMT models, and for TRANSFORMER we only keep the attention (with a single head) block while skipping other blocks such as resNet and layer normalization. More details can be found in the references (Vaswani et al., 2017).
To interpret RNN-SEARCH and TRANSFORMER, we define different  $\phi$  for them based on attention. For RNN-SEARCH, since attention is only defined on source side,  $\phi (w;y,c_t)$  can be defined only for the source words:
$$
\phi \left(x _ {i}; y, c _ {t}\right) = \alpha \left(s _ {t - 1}, h _ {i}\right)
$$
where  $\alpha$  is the attention weight defined in Eq.(3), and  $s_{t - 1}$  is the decoding state of RNN-SEARCH defined in Eq.(2). In contrast, TRANSFORMER defines the attention on both sides and thus  $\phi (w;y,c_t)$  is not constrained to source words:
$$
\phi (w; y, c _ {t}) = \left\{ \begin{array}{l l} \alpha (s _ {t + \frac {1}{2}}, h _ {i}) & \text {i f} w = x _ {i}, \\ \alpha (s _ {t - 1}, s _ {j}) & \text {i f} w = y _ {j} \text {a n d} j <   t, \end{array} \right.
$$
where  $s_{t-1}$  and  $s_{t+\frac{1}{2}}$  are defined in Eq.(4).
Gradient Different from attention that is restricted to a specific family of networks, the explanation methods based on gradient are more general. Suppose  $g(w,y)$  denotes the gradient of  $P(y\mid c_t)$  w.r.t to the variable  $w$  in  $c_{t}$ :
$$
g (w, y) = \frac {\partial P (y \mid c _ {t})}{\partial w} \tag {5}
$$
where  $\partial w$  denotes the gradient w.r.t the embedding of the word  $w$ , since a word itself is discrete and can not be taken gradient. Therefore,  $g(w,y)$  returns a vector with the same shape as the embedding of  $w$ . In this paper, we implement two different gradient-based explanation methods and derive different definitions of  $\phi (w;y,c_t)$  as follows.
Gradient Norm (Li et al., 2016): The first definition of  $\phi$  is the  $\ell - 1$  norm of  $g$ :
$$
\phi (w; y, c _ {t}) = | g (w, y) | _ {\ell - 1}.
$$
- Weighted Gradient (Ding et al., 2019): The second one is defined as the weighted sum of the embedding of  $w$ , with the return of  $g$  as the weight:
$$
\phi (w; y, c _ {t}) = g (w, y) ^ {\top} \cdot w.
$$
It is worth noting that for each sentence  $\langle \pmb{x},\pmb{y}\rangle$  one has to independently calculate  $\frac{\partial P(y|c_t)}{\partial w}$  for each timestep  $t$ . Therefore, one has to calculate  $|\pmb{y}|$  times of gradient for each sentence. In contrast, when training NMT, one only requires calculating sentence level gradient and it only calculates one gradient thanks to gradient accumulation in back propagation algorithm.
Prediction Difference Li et al. (2019) propose a prediction difference (PD) method, which defines the contribution of the word  $w$  by evaluating the change in the probability after removing  $w$  from  $c_{t}$ . Formally,  $\phi(w; y, c_{t})$  based on prediction difference is defined as follows:
$$
\phi (w; y, c _ {t}) = P (y \mid c _ {t}) - P (y \mid c _ {t} \backslash w)
$$
where  $P(y \mid c_t)$  is the NMT probability of  $y$  defined in Eq.(1), and  $P(y \mid c_t \backslash w)$  denotes the NMT probability of  $y$  after excluding  $w$  from its context  $c_t$ . To achieve the effect of excluding  $w$  from  $c_t$ , it simply replaces the word embedding of  $w$  with zero vector before feeding it into the NMT model.
3.1 Principled Metric
The key to our metric is described as follow: to define an explanation method  $\phi$  good enough in terms of our metric, the relevant words selected by  $\phi$  from the context  $c_{t}$  should have the potential to construct an optimal model that exhibits similar behavior to the target model  $P(y\mid c_t)$ . To formalize this metric, we first specify some necessary notations.
Assume that  $f(c_{t})$  is the target word predicted by  $P(y\mid c_t)$ , i.e.,  $f(c_{t}) = \arg \max_{y}P(y\mid c_{t})$ . In addition, let  $\mathcal{W}_{\phi}^{k}(c_{t})$  be the top-  $k$  relevant words on the source side and target side of the context  $c_{t}$ :
$$
\mathcal {W} _ {\phi} ^ {k} (c _ {t}) =
$$
$$
\operatorname {t o p} _ {w \in \boldsymbol {x}} ^ {k} \phi (w; f (c _ {t}), c _ {t}) \cup \operatorname {t o p} _ {w \in \boldsymbol {y} _ {<   t}} ^ {k} \phi (w; f (c _ {t}), c _ {t})
$$
where  $\cup$  denotes the union of two sets, and  $\mathrm{top}_{w\in \pmb{x}}^{k}\phi (w;f(c_{t}),c_{t})$  returns words corresponding to the  $k$  largest  $\phi$  values.
In addition, suppose  $Q(y \mid \mathcal{W}_{\phi}^{k}(c_{t}); \theta)$  ( $Q(\theta)$  or  $Q$  for brevity) is a proxy model that makes a translation decision on top of  $\mathcal{W}_{\phi}^{k}(c_{t})$  rather than the entire context  $c_{t}$  like a standard NMT model. Formally, we define a principled metric as follows:
Definition 1 The metric of  $\phi$  is defined by
$$
\min  _ {Q} \min  _ {\theta} - \mathbb {E} _ {c _ {t}} \left[ \log Q (f (c _ {t}) \mid \mathcal {W} _ {\phi} ^ {k} (c _ {t}); \theta) \right] \tag {6}
$$
2In fact,  $\mathcal{W}_{\phi}^{k}(c_{t})\to f(c_{t})$  can be considered as generalized translation rules obtained by  $\phi$  . In other words, the rules are extracted under teacher forcing decoding. In particular, if  $k = 1$  , this is similar to the statistical machine translation (SMT) with word level rules (Koehn, 2009), except that a generalized translation rule also involves a word from  $\pmb {y}_{< t}$  which simulates the role of language modeling in SMT.
where  $\mathbb{E}_{c_t}[\cdot ]$  denotes the expectation with respect to the data distribution of  $c_{t}$  , and  $Q$  is minimized over all possible proxy models.
The underlying idea of the above metric is to measure the expectation of the disagreement between an optimal proxy model  $Q$  constructed from  $\phi$  and the NMT model  $P$ . Here the disagreement is measured by the minus log-likelihood of  $Q$  over the data  $\langle \mathcal{W}_\phi^k (c_t),f(c_t)\rangle$  whose label  $f(c_{t})$  is generated from  $P$ .
Definition of Fidelity The metric of  $\phi$  actually defines fidelity by measuring how much the optimal proxy model defined on  $\mathcal{W}_{\phi}^{k}(c_{t})$  disagrees with  $P(y\mid c_t)$ . The mention of fidelity is widely used in model compression (Bucilua et al., 2006; Polino et al., 2018), model distillation (Hinton et al., 2015; Liu et al., 2018), and particularly in evaluating the explanation models for black-box neural networks (Lakkaraju et al., 2016; Bastani et al., 2017). These works focus on learning a specific model  $Q$  on which fidelity can be directly defined. However, we are interested in evaluating explanation methods  $\phi$  where  $Q$  is a latent variable that we have to minimize. By doing this, fidelity in our metric is defined on  $\phi$  as shown in Eq (6).
3.2 Approximation
Generally, it is intractable to exactly calculate the principled metric due to two main challenges. On one hand, the real data distribution of  $c_{t}$  is unknowable, making it impossible to exactly define the expectation with respect to an unknown distribution. On the other hand, the domain of a proxy model  $Q$  is not bounded, and it is difficult to minimize a model  $Q$  within an unbounded domain.
Empirical Risk Minimization Inspired by the statistical learning theory (Vapnik, 1999), we calculate the expected disagreement over  $c_t$  by a two-step strategy: we minimize the empirical risk to obtain an optimized  $\theta$  for a given  $Q$ ; and then we estimate the risk defined on a held-out test set by using the optimized  $\theta$ . In this way, we cast the principled metric into a standard machine learning task.
For a given model architecture  $Q$ , to optimize  $\theta$ , we first collect the training set as
$\{\langle \mathcal{W}_{\phi}^{k}(c_{t}),f(c_{t})\rangle \}$  for each sentence pair  $\langle \pmb {x},\pmb {y}\rangle$  at every time step  $t$  , where  $\langle x,y\rangle$  is a sentence pair from a given bilingual corpus  $\mathcal{D}_{\mathrm{train}} = \{\langle x^n,y^n\rangle |n = 1,\dots ,N\}$  . Then we optimize  $\theta$  by the empirical risk minimization:
$$
\min  _ {\theta} \sum_ {\langle \boldsymbol {x}, \boldsymbol {y} \rangle \in \mathcal {D} _ {\text {t r a i n}}} \sum_ {c _ {t}} - \log Q (f (c _ {t}) \mid \mathcal {W} _ {\phi} ^ {k} (c _ {t}); \theta) \tag {7}
$$
Proxy Model Selection In response to the second challenge of the unbounded domain, we define a surrogate distribution family  $\mathcal{Q}$ , and then approximately calculate Eq.(6) within  $\mathcal{Q}$  instead:
$$
\min  _ {Q \in \mathcal {Q}} \min  _ {\theta} - \mathbb {E} _ {c _ {t}} \left[ \log Q (f (c _ {t}) \mid \mathcal {W} _ {\phi} ^ {k} (c _ {t}); \theta) \right] \tag {8}
$$
We consider three different proxy models including multi-layer feedforward network (FN), recurrent network (RN) and self-attention network (SA). In details, for different networks  $\epsilon \in \{\mathrm{FN},\mathrm{RN},\mathrm{SA}\}$ , the proxy model  $Q^{\epsilon}$  is defined as follows:
$$
Q ^ {\epsilon} (y \mid \mathcal {W} _ {\phi} ^ {k} (c _ {t})) = P (y \mid s _ {t} ^ {\epsilon})
$$
where  $s_t^\epsilon$  is the decoding state regarding different architecture  $\epsilon$ . Specifically, for feedforward network, the decoding state is defined by
$$
s _ {t} ^ {\mathrm {F N}} = \mathrm {F N N} (\tilde {x} _ {1}, \dots , \tilde {x} _ {k}, \tilde {y} _ {1}, \dots , \tilde {y} _ {k}).
$$
For  $\epsilon \in \{\mathrm{RN},\mathrm{SA}\}$ , the decoding state  $s_t^\epsilon$  is defined by
$$
s _ {t} ^ {\epsilon} = \mathrm {A t t n} \big (s _ {0}, \{h _ {\tilde {x} _ {1}}, \dots , h _ {\tilde {x} _ {k}}, h _ {\tilde {y} _ {1}} \dots , h _ {\tilde {y} _ {k}} \} \big),
$$
where  $\tilde{x}$  and  $\tilde{y}$  are source and target side words from  $\mathcal{W}_{\phi}^{k}(c_{t})$ ,  $s_0$  is the query of init state,  $h$  is the position-aware representations of words, generated by the encoder of RN or SA as defined in Eq.(3) and Eq.(4). For RN,  $s_t^{\mathrm{RN}}$  is the weight-sum vectors of a bidirectional LSTM over all selected top  $k$  source and target words; while for SA,  $s_t^{\mathrm{SA}}$  is the weight-sum of vectors over the SA networks.
3.3 Evaluation Paradigm
Given a bilingual training set  $\mathcal{D}_{\mathrm{train}}$  and a bilingual test set  $\mathcal{D}_{\mathrm{test}}$ , we evaluate an explanation method  $\phi$  w.r.t the NMT model  $P(y \mid c_t)$  by setting the proxy model family  $\mathcal{Q}(\theta)$  to include three neural networks as defined before. Following the
It is natural to extend our definition by using other similar disagreement measures such as the KL distance. Since the KL distance requires additional GPU memory to restore the distribution  $P$  in the implementation, we employ the minus log-likelihood for efficiency in our experiments.
standard process of addressing a machine learning problem, Algorithm 1 summarizes the procedure to approximately calculate the metric of  $\phi$  on the test dataset  $\mathcal{D}_{\mathrm{test}}$ , which returns the preplexity (PPL) on  $\mathcal{F}\mathcal{W}_{\mathrm{test}}$ .<sup>4</sup>
In this paper, we try four different choices to specify the surrogate family, i.e.,  $\mathcal{Q} = \{Q^{\mathrm{FN}}\}$ ,  $\mathcal{Q} = \{Q^{\mathrm{RN}}\}$ ,  $\mathcal{Q} = \{Q^{\mathrm{SA}}\}$ , and  $\mathcal{Q} = \{Q^{\mathrm{FN}}, Q^{\mathrm{RN}}, Q^{\mathrm{SA}}\}$ , leading to four instances of our metric respectively denoted as FN, RN, SA and Comb. In addition, as the baseline metric, we employ the well-trained NMT model  $P$  as the proxy model  $Q$  by masking out the input words that do not appear in the rule set  $\mathcal{W}_{\phi}^{k}(c_{t})$ . For the baseline metric, it doesn't require to train  $Q's$  parameter  $\theta$  and tests on  $\mathcal{D}_{\mathrm{test}}$  only. Since  $P$  is trained with the entire context  $c_{t}$  whereas it is testified on  $\mathcal{W}_{\phi}^{k}(c_{t})$ , this mismatch may lead to poor performance and is thus less trusted. This baseline metric extends the idea of Arras et al. (2016); Denil et al. (2014) from classification tasks to structured prediction tasks like machine translation which are highly dependent on context rather than just keywords.
In this section, we conduct experiments to prove the effectiveness of our metric from two viewpoints: how good an explanation method is and
which explanation method is better than others.
Datasets We carry out our experiments on three standard IWSLT translation tasks including IWSLT14  $\mathrm{De} \Rightarrow \mathrm{En}$  (167k sentence pairs), IWSLT17  $\mathrm{Zh} \Rightarrow \mathrm{En}$  (237k sentence pairs) and IWSLT17  $\mathrm{Fr} \Rightarrow \mathrm{En}$  (229k sentence pairs). All these datasets are tokenized and applied BPE (Byte-Pair Encoding) following Ott et al. (2019). The target side vocabulary sizes of the three datasets are 8876, 11632, and 9844 respectively. In addition, we carry out extended experiments on three largescale WMT translation tasks including WMT14  $\mathrm{De} \Rightarrow \mathrm{En}$  (4.5m sentence pairs), WMT17  $\mathrm{Zh} \Rightarrow \mathrm{En}$  (22m sentence pairs) and WMT14  $\mathrm{Fr} \Rightarrow \mathrm{En}$  (40.8m sentence pairs), with vocabulary sizes 22568, 29832, 27168 respectively.
NMT Systems To examine the generality of our evaluation method, we conduct experiments on two NMT systems, i.e. RNN-SEARCH (denoted by RNN) and TRANSFORMER (denoted by Trans.), both of which are implemented with fairseq (Ott et al., 2019). For RNN, we adopt the 1-layer RNN with LSTM cells whose encoder (bi-directional) and decoder hidden units are 256 and 512 respectively. For TRANSFORMER on the IWSLT datasets, the number of layers and attention heads are 2 and 4 respectively. For both models, we set the embedding dimensions as 256. On WMT datasets, we simply use TRANSFORMER-BASE with 4 attention heads. The performances of our NMT models are comparable to those reported in recent literature (Tan et al., 2019).
Explanation Methods On both NMT systems, we implement four explanation methods, i.e. Attention (ATTN), gradient norm (NGRAD), weighted gradient (WGRAD), and prediction difference (PD) as mentioned in Section §2.
Our metric We implemented five instantiations of the proposed metric including FN, RN, SA, Comb, and Baseline (Base for brevity) as presented in section §3.3. To configure them, we adopt the same settings from NMT systems to train SA and RN. FN is implemented with feeding the features of bag of words through a 3-layer fully connected network. As given in algorithm 1, the approximate fidelity is estimated through  $Q$  with the lowest PPL, therefore the best metric is
Note that the negative log-likelihood in Eq. 6 is proportional to PPL and thus we use PPL as the metric value in this paper.
NMT	Metric	ATTN	PD	NGRAD	WGRAD
Trans	Base	196.9	54.3	193.4	13400
FN	13.9	5.8	11.3	131.2
RN	13.8	5.7	10.7	126.7
SA	13.9	5.5	10.8	119.5
Comb	13.8	5.5	10.7	119.5
RNN	Base	-	54.2	90.3	28587
FN	-	6.7	8.3	170.8
RN	-	6.5	7.8	163.2
SA	-	6.5	8.1	154.9
Comb	-	6.5	7.8	154.9
that achieves the lowest PPL since it results in a closer approximation to the real fidelity.
4.2 Experiments on IWSLT tasks
In this subsection, we first conduct experiments and analysis on the IWSLT  $\mathrm{De} \Rightarrow \mathrm{En}$  task to configure fidelity-based metric and then extend the experiments to other IWSLT tasks.