SlowGuess/NAACL_2021_OCR huggingface dataset:
Ben Peters† and André F. T. Martins†‡*  
†Instituto de Telecomunicações, Instituto Superior Tecnico, Lisbon, Portugal  
‡LULIS (Lisbon ELLIS Unit), Lisbon, Portugal  
*Unbabel, Lisbon, Portugal  
benzurdopeters@gmail.com, andre.t.martins@tecnico.ulisboa.pt
Current sequence-to-sequence models are trained to minimize cross-entropy and use softmax to compute the locally normalized probabilities over target sequences. While this setup has led to strong results in a variety of tasks, one unsatisfying aspect is its length bias: models give high scores to short, inadequate hypotheses and often make the empty string the argmax—the so-called cat got your tongue problem. Recently proposed entmax-based sparse sequence-to-sequence models present a possible solution, since they can shrink the search space by assigning zero probability to bad hypotheses, but their ability to handle word-level tasks with transformers has never been tested. In this work, we show that entmax-based models effectively solve the cat got your tongue problem, removing a major source of model error for neural machine translation. In addition, we generalize label smoothing, a critical regularization technique, to the broader family of Fenchel-Young losses, which includes both cross-entropy and the entmax losses. Our resulting label-smoothed entmax loss models set a new state of the art on multilingual grapheme-to-phoneme conversion and deliver improvements and better calibration properties on cross-lingual morphological inflection and machine translation for 7 language pairs.
Sequence-to-sequence models (seq2seq: Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017) have become a powerful and flexible tool for a variety of NLP tasks, including machine translation (MT), morphological inflection (MI; Faruqui et al., 2016), and grapheme-to-phoneme conversion (G2P; Yao and Zweig, 2015). These models often perform well, but they have a bias that favors short hypotheses. This bias is problematic: it has been pointed out as the cause (Koehn and Knowles, 2017; Yang et al., 2018; Murray and Chi
ang, 2018) of the beam search curse, in which increasing the width of beam search actually decreases performance on neural machine translation (NMT). Further illustrating the severity of the problem, Stahlberg and Byrne (2019) showed that the highest-scoring target sequence in NMT is often the empty string, a phenomenon they dubbed the cat got your tongue problem. These results are undesirable because they show that NMT models' performance depends on the search errors induced by a narrow beam. It would be preferable for models to assign higher scores to good translations than to bad ones, rather than to depend on search errors to make up for model errors.
The most common way to alleviate this shortcoming is by altering the decoding objective (Wu et al., 2016; He et al., 2016; Yang et al., 2018; Meister et al., 2020a), but this does not address the underlying problem: the model overestimates the probability of implausible hypotheses. Other solutions use alternate training strategies (Murray and Chiang, 2018; Shen et al., 2016), but it would be preferable not to change the training algorithm.
In this paper, we propose a solution based on sparse seq2seq models (Peters et al., 2019), which replace the output softmax (Bridle, 1990) with the entmax transformation. Entmax, unlike softmax, can learn locally sparse distributions over the target vocabulary. This allows a sparse model to shrink the search space: that is, it can learn to give inadequate hypotheses zero probability, instead of counting on beam search to prune them. This has already been demonstrated for MI, where the set of possible hypotheses is often small enough to make beam search exact (Peters et al., 2019; Peters and Martins, 2019). We extend this analysis to MT: although exact beam search is not possible for this large vocabulary task, we show that entmax models prune many inadequate hypotheses, effectively solving the cat got your tongue problem.
Despite this useful result, one drawback of ent
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2642-2654. June 6-11, 2021. ©2021 Association for Computational Linguistics
max is that it is not compatible with label smoothing (Szegedy et al., 2016), a useful regularization technique that is widely used for transformers (Vaswani et al., 2017). We solve this problem by generalizing label smoothing from the cross-entropy loss to the wider class of Fenchel-Young losses (Blondel et al., 2020), which includes the entmax loss as a particular case. We show that combining label smoothing with entmax loss improves results on both character- and word-level tasks while keeping the model sparse. We note that, although label smoothing improves calibration, it also exacerbates the cat got your tongue problem regardless of loss function.
To sum up, we make the following contributions:
- We show empirically that models trained with entmax loss rarely assign nonzero probability to the empty string, demonstrating that entmax loss is an elegant way to remove a major class of NMT model errors.
- We generalize label smoothing from the cross-entropy loss to the wider class of Fenchel-Young losses, exhibiting a formulation for label smoothing which, to our knowledge, is novel.
- We show that Fenchel-Young label smoothing with entmax loss is highly effective on both character- and word-level tasks. Our technique allows us to set a new state of the art on the SIGMORPHON 2020 shared task for multilingual G2P (Gorman et al., 2020). It also delivers improvements for crosslingual MI from SIGMORPHON 2019 (McCarthy et al., 2019) and for MT on IWSLT 2017 German  $\leftrightarrow$  English (Cettolo et al., 2017), KFTT Japanese  $\leftrightarrow$  English (Neubig, 2011), WMT 2016 Romanian  $\leftrightarrow$  English (Bojar et al., 2016), and WMT 2014 English  $\rightarrow$  German (Bojar et al., 2014) compared to smoothed and unsmoothed cross-entropy loss.
A seq2seq model learns a probability distribution  $p_{\theta}(y \mid x)$  over sequences  $y$  from a target vocabulary  $V$ , conditioned on a source sequence  $x$ . This
distribution is then used at decoding time to find the most likely sequence  $\hat{y}$ :
$$
\hat {y} = \underset {y \in V ^ {*}} {\operatorname {a r g m a x}} p _ {\boldsymbol {\theta}} (y \mid x), \tag {1}
$$
where  $V^{*}$  is the Kleene closure of  $V$ . This is an intractable problem; seq2seq models depend on heuristic search strategies, most commonly beam search (Reddy et al., 1977). Most seq2seq models are locally normalized, with probabilities that decompose by the chain rule:
$$
p _ {\boldsymbol {\theta}} (y \mid x) = \prod_ {i = 1} ^ {| y |} p _ {\boldsymbol {\theta}} \left(y _ {i} \mid x, y _ {<   i}\right). \tag {2}
$$
This factorization implies that the probability of a hypothesis being generated is monotonically nonincreasing in its length, which favors shorter sequences. This phenomenon feeds the beam search curse because short hypotheses<sup>2</sup> are pruned from a narrow beam but survive a wider one.
The conditional distribution  $p_{\pmb{\theta}}(y_i \mid x, y_{<i})$  is obtained by first computing a vector of scores (or "logits")  $z = f_{\pmb{\theta}}(x, y_{<i}) \in \mathbb{R}^{|V|}$ , where  $f_{\pmb{\theta}}$  is parameterized by a neural network, and then applying a transformation  $\pi: \mathbb{R}^{|V|} \to \triangle^{|V|}$ , which maps scores to the probability simplex  $\triangle^{|V|} := \{ \pmb{p} \in \mathbb{R}^{|V|}: \pmb{p} \geq 0, \| \pmb{p} \|_1 = 1 \}$ . The usual choice for  $\pi$  is softmax (Bridle, 1990), which returns strictly positive values, ensuring that all sequences  $\in V^*$  have nonzero probability. Coupled with the short sequence bias, this causes significant model error.
Sparse seq2seq models. In a sparse model, the output softmax is replaced by a transformation  $\pi$  from the entmax family (Peters et al., 2019). Like softmax, entmax transformations return a vector in the simplex and are differentiable (almost) everywhere. However, unlike softmax, they are capable of producing sparse probability distributions. Concretely, this is done by using the so-called “ $\beta$ -exponential function” (Tsallis, 1988) in place of the exponential, where  $\beta \geq 0$ :
$$
\exp_ {\beta} (v) := \left\{ \begin{array}{l l} {[ 1 + (1 - \beta) v ] _ {+} ^ {1 / (1 - \beta)},} & {\beta \neq 1} \\ {\exp (v),} & {\beta = 1.} \end{array} \right. \tag {3}
$$
The  $\beta$ -exponential function converges to the regular exponential when  $\beta \to 1$ . Entmax models assume that  $p(y_{i} \mid x, y_{<i})$  results from an  $\alpha$ -entmax
2We use "hypothesis" to mean any sequence that ends with the special end-of-sequence token.
1Our code is available at https://github.com/deep-spin/S7.
transformation of the scores  $z$ , defined as
$$
[ \alpha - \operatorname {e n t m a x} (z) ] _ {y} := \exp_ {2 - \alpha} \left(z _ {y} - \tau_ {\alpha} (z)\right), \tag {4}
$$
where  $\tau_{\alpha}(z)$  is a constant which ensures normalization. When  $\alpha = 1$ , (4) turns to a regular exponential function and  $\tau_1(z) = \log \sum_{y' = 1}^{|V|}\exp (z_{y'})$  is the log-partition function, recovering softmax. When  $\alpha = 2$ , we recover sparsemax (Martins and Astudillo, 2016). For  $\alpha \in \{1.5,2\}$ , fast algorithms to compute (4) are available which are almost as fast as evaluating softmax. For other values of  $\alpha$ , slower bisection algorithms exist.
Entmax transformations are sparse for any  $\alpha > 1$ , with higher values tending to produce sparser outputs. This sparsity allows a model to assign exactly zero probability to implausible hypotheses. For tasks where there is only one correct target sequence, this often allows the model to concentrate all probability mass into a small set of hypotheses, making search exact (Peters and Martins, 2019). This is not possible for open-ended tasks like machine translation, but the model is still locally sparse, assigning zero probability to many hypotheses. These hypotheses will never be selected at any beam width.
Fenchel-Young Losses. Inspired by the softmax generalization above, Blondel et al. (2020) provided a tool for constructing a convex loss function. Let  $\Omega : \triangle^{|V|} \to \mathbb{R}$  be a strictly convex regularizer which is symmetric, i.e.,  $\Omega(\Pi p) = \Omega(p)$  for any permutation  $\Pi$  and any  $p \in \triangle^{|V|}$ .3 Equipped with  $\Omega$ , we can define a regularized prediction function  $\hat{\pi}_{\Omega}: \mathbb{R}^{|V|} \to \triangle^{|V|}$ , with this form:
$$
\hat {\pi} _ {\Omega} (\boldsymbol {z}) = \underset {\boldsymbol {p} \in \triangle^ {| V |}} {\operatorname {a r g m a x}} \boldsymbol {z} ^ {\top} \boldsymbol {p} - \Omega (\boldsymbol {p}) \tag {5}
$$
where  $\pmb{z} \in \mathbb{R}^{|V|}$  is the vector of label scores (logits) and  $\Omega: \triangle^{|V|} \to \mathbb{R}$  is a regularizer. Equation 5 recovers both softmax and entmax with particular choices of  $\Omega$ : the negative Shannon entropy,  $\Omega(\pmb{p}) = \sum_{y \in V} p_y \log p_y$ , recovers the variational form of softmax (Wainwright and Jordan, 2008), while the negative Tsallis entropy (Tsallis, 1988) with parameter  $\alpha$ , defined as
$$
\Omega_ {\alpha} (\boldsymbol {p}) = \left\{ \begin{array}{l l} \frac {1}{\alpha (\alpha - 1)} \left(\sum_ {y \in V} p _ {y} ^ {\alpha} - 1\right), & \text {i f} \alpha \neq 1 \\ \sum_ {y \in V} p _ {y} \log p _ {y}, & \text {i f} \alpha = 1, \end{array} \right. \tag {6}
$$
Figure 1: Diagram illustrating Fenchel-Young losses and the particular case of  $\alpha$ -entmax family. The case  $\alpha = 1$  corresponds to softmax and the cross-entropy loss,  $\alpha = 2$  to the sparsemax loss, and  $\alpha = 1.5$  to the 1.5-entmax loss. Any choice of  $\alpha > 1$  (in blue) can lead to sparse distributions.
recovers the  $\alpha$ -entmax transformation in (4), as shown by Peters et al. (2019).
Given the choice of  $\Omega$ , the Fenchel-Young loss function  $L_{\Omega}$  is defined as
$$
L _ {\Omega} (\boldsymbol {z}; \boldsymbol {q}) := \Omega^ {*} (\boldsymbol {z}) + \Omega (\boldsymbol {q}) - \boldsymbol {z} ^ {\top} \boldsymbol {q}, \tag {7}
$$
where  $\pmb{q}$  is a target distribution, most commonly a one-hot vector indicating the gold label,  $\pmb{q} = e_{y^*} = [0, \dots, 0, \underbrace{1}_{y^* \text{-th entry}}, 0, \dots, 0]$ , and  $\Omega^*$  is the
convex conjugate of  $\Omega$ , defined variationally as:
$$
\Omega^ {*} (\boldsymbol {z}) := \max  _ {\boldsymbol {p} \in \triangle | V |} \boldsymbol {z} ^ {\top} \boldsymbol {p} - \Omega (\boldsymbol {p}). \tag {8}
$$
The name stems from the Fenchel-Young inequality, which states that the quantity (7) is nonnegative (Borwein and Lewis, 2010, Prop. 3.3.4). When  $\Omega$  is the generalized negative entropy, the loss (7) becomes the Kullback-Leibler divergence between  $q$  and  $\mathrm{softmax}(z)$  (KL divergence; Kullback and Leibler, 1951), which equals the cross-entropy when  $q$  is a one-hot vector. More generally, if  $\Omega \equiv \Omega_{\alpha}$  is the negative Tsallis entropy (6), we obtain the  $\alpha$ -entmax loss (Peters et al., 2019).
Fenchel-Young losses have nice properties for training neural networks with backpropagation: they are non-negative, convex, and differentiable as long as  $\Omega$  is strictly convex (Blondel et al., 2020, Prop. 2). Their gradient is
$$
\nabla_ {\boldsymbol {z}} L _ {\Omega} (\boldsymbol {z}; \boldsymbol {q}) = \hat {\pi} _ {\Omega} (\boldsymbol {z}) - \boldsymbol {q}, \tag {9}
$$
which generalizes the gradient of the cross-entropy loss. Figure 1 illustrates particular cases of Fenchel-Young losses considered in this paper.
3 Fenchel-Young Label Smoothing
Label smoothing (Szegedy et al., 2016) has become a popular technique for regularizing the output of
<sup>3</sup>It is instructive to think of  $\Omega$  as a generalized negative entropy: for example, as shown in Blondel et al. (2020, Prop. 4), strict convexity and symmetry imply that  $\Omega$  is minimized by the uniform distribution. For a more comprehensive treatment of Fenchel-Young losses, see the cited work.
a neural network. The intuition behind it is that using the gold target labels from the training set can lead to overconfident models. To overcome this, label smoothing redistributes probability mass from the gold label to the other target labels. When the redistribution is uniform, Pereyra et al. (2017) and Meister et al. (2020b) pointed out that this is equivalent (up to scaling and adding a constant) to adding a second term to the loss that computes the KL divergence  $D_{\mathrm{KL}}(\pmb{u} \| \pmb{p}_{\theta})$  between a uniform distribution  $\pmb{u}$  and the model distribution  $\pmb{p}_{\theta}$ . While it might seem appealing to add a similar KL regularizer to a Fenchel-Young loss, this is not possible when  $\pmb{p}_{\theta}$  contains zeroes because the KL divergence term becomes infinite. This makes vanilla label smoothing incompatible with sparse models. Fortunately, there is a more natural generalization of label smoothing to Fenchel-Young losses. For  $\epsilon \in [0, 1]$ , we define the Fenchel-Young label smoothing loss as follows:
$$
L _ {\Omega , \epsilon} (\boldsymbol {z}, \boldsymbol {e} _ {y ^ {*}}) := L _ {\Omega} (\boldsymbol {z}, (1 - \epsilon) \boldsymbol {e} _ {y ^ {*}} + \epsilon \boldsymbol {u}). \tag {10}
$$
The intuition is the same as in cross-entropy label smoothing: the target one-hot vector is mixed with a uniform distribution.
This simple definition leads to the following result, proved in Appendix A:
Proposition 1. The Fenchel-Young label smoothing loss can be written as
$$
L _ {\Omega , \epsilon} (\boldsymbol {z}, \boldsymbol {e} _ {y ^ {*}}) = L _ {\Omega} (\boldsymbol {z}, \boldsymbol {e} _ {y ^ {*}}) + \epsilon \left(z _ {y ^ {*}} - \bar {z}\right) + C, \tag {11}
$$
where  $C = -\Omega (e_{y^*}) + \Omega ((1 - \epsilon)e_{y^*} + \epsilon u)$  is a constant which does not depend on  $\pmb{z}$ , and  $\bar{z} := \pmb{u}^\top \pmb{z}$  is the average of the logits. Furthermore, up to a constant, we also have
$$
L _ {\Omega , \epsilon} (\boldsymbol {z}, \boldsymbol {e} _ {y ^ {*}}) \propto L _ {\Omega} (\boldsymbol {z}, \boldsymbol {e} _ {y ^ {*}}) + \lambda L _ {\Omega} (\boldsymbol {z}, \boldsymbol {u}), \tag {12}
$$
where  $\lambda = \frac{\epsilon}{1 - \epsilon}$
The first expression (11) shows that, up to a constant, the smoothed Fenchel-Young loss equals the original loss plus a linear regularizer  $\epsilon (z_{y^{*}} - \bar{z})$ . While this regularizer can be positive or negative, we show in Appendix A that its sum with the original loss  $L_{\Omega}(\pmb {z},\pmb{e}_{y^*})$  is always non-negative – intuitively, if the score  $z_{y^{*}}$  is below the average, resulting in negative regularization, the unregularized loss will also be larger, and the two terms balance each other. Figure 2 shows the effect of this regularization in the graph of the loss – we see that a
correct prediction is linearly penalized with a slope of  $\epsilon$ ; the larger the confidence, the larger the penalty. In particular, when  $\Omega$  is the Shannon negentropy, this result shows a simple expression for vanilla label smoothing which, to the best of our knowledge, is novel. The second expression (12) shows that it can also be seen as a form of regularization towards the uniform distribution. When  $-\Omega$  is the Shannon entropy, the regularizer becomes a KL divergence and we obtain the interpretation of label smoothing for cross-entropy provided by Pereyra et al. (2017) and Meister et al. (2020b). Therefore, the same interpretation holds for the entire Fenchel-Young family if the regularization uses the corresponding Fenchel-Young loss with respect to a uniform.
Gradient of Fenchel-Young smoothed loss. From Prop. 1 and Equation 9, we immediately obtain the following expression for the gradient of the smoothed loss:
$$
\begin{array}{l} \nabla_ {\boldsymbol {z}} L _ {\Omega , \epsilon} (\boldsymbol {z}, \boldsymbol {e} _ {y ^ {*}}) = \\ = \nabla_ {\boldsymbol {z}} L _ {\Omega} (\boldsymbol {z}, \boldsymbol {e} _ {y ^ {*}}) + \epsilon (\boldsymbol {e} _ {y ^ {*}} - \boldsymbol {u}) \\ = \hat {\pi} _ {\Omega} (z) - (1 - \epsilon) e _ {y ^ {*}} - \epsilon u, \tag {13} \\ \end{array}
$$
that is, the computation of this gradient is straightforward by adding a constant vector to the original gradient of the Fenchel-Young loss; as the latter, it only requires the ability of computing the  $\hat{\pi}_{\Omega}$  transformation, which is efficient in the entmax case as shown by Peters et al. (2019). Note that, unlike the gradient of the original entmax loss, the gradient of its smoothed version is not sparse (in the sense that it will not contain many zeroes); however, since  $\pmb{u}$  is the uniform distribution, it will contain many constant terms with value  $-\epsilon / |V|$ .
We trained seq2seq models for three tasks: multilingual G2P, crosslingual MI, and MT. These tasks present very different challenges. In G2P and MI, character-level vocabularies are small and there is usually only one correct target sequence. The relative simplicity of these tasks is offset by the small quantity of training data and the strict evaluation: the model must produce exactly the right sequence. This tests Fenchel-Young label smoothing's ability to learn exactly in a low-resource setting. On the other hand, MT is trained with much larger corpora and evaluated with less strict metrics, but uses subword vocabularies with sizes in the tens of thousands and has to manage more ambiguity because sentences typically have many correct translations.
Figure 2: Fenchel-Young label smoothing with various  $\epsilon$  values in the two-dimensional case for the cross-entropy loss (left) and sparsemax loss (right). In both cases we define  $s = z_2 - z_1$  and assume  $y^{*} = 1$ .
In all tasks, we vary two hyperparameters:
- Entmax Loss  $\alpha$ : this influences the sparsity of the probability distributions the model returns, with  $\alpha = 1$  recovering cross-entropy and larger  $\alpha$  values encouraging sparser distributions. We use  $\alpha \in \{1, 1.5, 2\}$  for G2P and MI, and  $\alpha \in \{1, 1.5\}$  for MT.
- Fenchel-Young Label Smoothing  $\epsilon$ : higher values give more weight to the uniform smoothing distribution, discouraging sparsity. We use  $\epsilon \in \{0, 0.01, 0.02, \dots, 0.15\}$  for G2P,  $\epsilon \in \{0, 0.01, 0.05, 0.1\}$  for MI, and  $\epsilon \in \{0, 0.01, 0.1\}$  for MT.
We trained all models with early stopping for a maximum of 30 epochs for MI, 15 epochs for WMT 2014 English  $\rightarrow$  German MT, and 100 epochs otherwise, keeping the best checkpoint according to a task-specific validation metric: Phoneme Error Rate for G2P, average Levenshtein distance for MI, and detokenized BLEU score for MT. At test time, we decoded with a beam width of 5. Our PyTorch code (Paszke et al., 2017) is based on JoeyNMT (Kreutzer et al., 2019) and the entmax implementation from the entmax package.
4.1 Multilingual G2P
Data. We use the data from SIGMORPHON 2020 Task 1 (Gorman et al., 2020), which includes 3600 training examples in each of 15 languages. We train a single multilingual model (following Peters and Martins, 2020) which must learn to apply spelling rules from several writing systems.
Training. Our models are similar to Peters and Martins (2020)'s RNNs, but with entmax 1.5 attention, and language embeddings only in the source.
Results. Multilingual G2P results are shown in Table 1, along with the best previous result (Yu et al., 2020). We report two error metrics, each of which is computed per-language and averaged:
- Word Error Rate (WER) is the percentage of hypotheses which do not exactly match the reference. This harsh metric gives no credit for partial matches.
- Phoneme Error Rate (PER) is the sum of Levenshtein distances between each hypothesis and the corresponding reference, divided by the total length of the references.
These results show that the benefits of sparse losses and label smoothing can be combined. Individually, both label smoothing and sparse loss functions  $(\alpha > 1)$  consistently improve over unsmoothed cross-entropy  $(\alpha = 1)$ . Together, they produce the best reported result on this dataset. Our approach is very simple, as it requires manipulating only the loss function: there are no changes to the standard seq2seq training or decoding algorithms, no language-specific training or tuning, and no external auxiliary data. In contrast, the previous state of the art (Yu et al., 2020) relies on a complex self-training procedure in which a genetic algorithm is used to learn to ensemble several base models.
4.2 Crosslingual MI
Data. Our data come from SIGMORPHON 2019 Task 1 (McCarthy et al., 2019), which includes datasets for 100 language pairs. Each training set combines roughly 10,000 examples from a high resource language with 100 examples from a (simulated) low resource language. Development and test sets only cover the low resource language.
5 Although most of the low resource sets are for languages that lack real-world NLP resources, others are simply small training sets in widely-spoken languages such as Russian.
4https://github.com/deep-spin/entmax
α	ε	Single	Ensemble
WER ↓	PER ↓	WER ↓	PER ↓
1	0	18.14 ± 2.87	3.95 ± 1.24	14.74	2.96
0.15	15.55 ± 0.48	3.09 ± 0.10	13.87	2.77
1.5	0	15.25 ± 0.25	3.05 ± 0.03	13.79	2.77
0.04	14.18 ± 0.24	2.86 ± 0.05	13.47	2.69
2	0	15.08 ± 0.28	3.04 ± 0.06	13.84	2.75
0.04	14.17 ± 0.20	2.88 ± 0.04	13.51	2.73
(Yu et al., 2020)	13.81	2.76
Table 1: Multilingual G2P results on the SIGMORPHON 2020 Task 1 test set, macro-averaged across languages. Numbers in the Single column are the average of five independent model runs. The same models are used for the ensembles. Note that  $\alpha = 1$  and  $\alpha = 2$  recover cross-entropy and sparsemax loss, respectively.
α	ε	Acc. ↑	Lev. Dist. ↓
1	0	50.16	1.12
>0	52.72	1.04
1.5	0	55.21	0.98
>0	57.40	0.92
2	0	56.01	0.97
>0	57.77	0.90
CMU-03	58.79	1.52
Table 2: Macro-averaged MI results on the SIGMOR-PHON 2019 Task 1 test set. When  $\epsilon > 0$ , it is tuned separately for each language pair.