SlowGuess/NAACL_2022_OCR huggingface dataset:
$$
\sigma \Phi^ {- 1} (1 - p) = \sigma m (p) - \sqrt {4 \cdot \frac {\epsilon}{p} - \sigma^ {2} s (p)} \quad \Leftrightarrow \quad \Phi^ {- 1} (1 - p) = m (p) - \sqrt {\frac {4}{\sigma^ {2}} \cdot \frac {\epsilon}{p} - s (p)}. \tag {12}
$$
Let  $p^*$  be the solution of (12). Then, taking into account (10), we have
$$
\tau (\mathbf {z}) \approx \frac {\sigma m (p ^ {*})}{2} - \sqrt {\frac {\epsilon}{p ^ {*}} - \frac {\sigma^ {2} s (p ^ {*})}{4}} = \frac {\sigma}{2} \left(m (p ^ {*}) - \sqrt {\frac {4}{\sigma^ {2}} \cdot \frac {\epsilon}{p ^ {*}} - s (p ^ {*})}\right) = \frac {\sigma}{2} \Phi^ {- 1} (1 - p ^ {*}),
$$
Lemma 3. Let  $\mathbf{z} = \mathbf{W}\mathbf{x}$  be a pre-softmax vector of logits in the OpenNMT-py (Klein et al., 2017) implementation of the Transformer model (Vaswani et al., 2017). Then for any input, in a non-trained model the logits  $z_{1},\ldots ,z_{d}$  are distributed according to the normal distribution  $\mathcal{N}\left(0,\frac{2\cdot d_{model}}{d_{model} + d_{vocab}}\right)$ , where  $d_{model}$  is the size of hidden representations, and  $d_{vocab}$  is the vocabulary size for a target language.
Proof. The default Transformer configuration in OpenNMT-py implies that the elements  $w_{ij}$  of  $\mathbf{W}$  are initialized from a uniform distribution  $\mathcal{U}[-a, a]$ , where  $a = \sqrt{\frac{6}{d_{\mathrm{model}} + d_{\mathrm{vocab}}}}$ , thus
$$
\mathbb {E} \left[ w _ {i j} \right] = 0, \quad \operatorname {V a r} \left[ w _ {i j} \right] = \frac {(2 a) ^ {2}}{1 2} = \frac {a ^ {2}}{3} = \frac {2}{d _ {\mathrm {m o d e l}} + d _ {\mathrm {v o c a b}}} \tag {13}
$$
Since  $\mathbf{x}$  is the result of a layer normalization (Ba et al., 2016), we have
$$
\frac {1}{d _ {\mathrm {m o d e l}}} \sum_ {j = 1} ^ {d _ {\mathrm {m o d e l}}} x _ {j} = 0, \quad \frac {1}{d _ {\mathrm {m o d e l}}} \sum_ {j = 1} ^ {d _ {\mathrm {m o d e l}}} x _ {j} ^ {2} = 1 \tag {14}
$$
Therefore, from (13) and (14), we have
$$
\mathbb {E} \left[ z _ {i} \right] = \mathbb {E} \left[ \sum_ {j = 1} ^ {d _ {\mathrm {m o d e l}}} w _ {i j} x _ {j} \right] = \sum_ {j = 1} ^ {d _ {\mathrm {m o d e l}}} \mathbb {E} \left[ w _ {i j} \right] \cdot x _ {j} = 0,
$$
$$
\mathrm {V a r} [ z _ {i} ] = \mathrm {V a r} \left[ \sum_ {j = 1} ^ {d _ {\mathrm {m o d e l}}} w _ {i j} x _ {j} \right] = \frac {2}{d _ {\mathrm {m o d e l}} + d _ {\mathrm {v o c a b}}} \sum_ {j = 1} ^ {d _ {\mathrm {m o d e l}}} x _ {j} ^ {2} = \frac {2 \cdot d _ {\mathrm {m o d e l}}}{d _ {\mathrm {m o d e l}} + d _ {\mathrm {v o c a b}}}.
$$
Being a sum of independent random variables, by the Central Limit Theorem, each  $z_{i}$  tends to normal distribution with the mean and variance above.
B.3 Derivation of the Equation (5)
We provide derivation for the case of  $\alpha$ -ReLU. Extension to  $\alpha$ -entmax and softmax is done analogously. Let  $\mathbf{x} \in \mathbb{R}^{n_0}$  be the input vector. We define a feedforward neural network with  $L - 1$  hidden layers recursively:
$$
\mathbf {h} ^ {(0)} = \mathbf {x}
$$
$$
\mathbf {z} ^ {(k)} = \frac {1}{\sqrt {n _ {k - 1}}} \mathbf {W} ^ {(k - 1)} \mathbf {h} ^ {(k - 1)},
$$
$$
\mathbf {h} ^ {(k)} = \sigma (\mathbf {z} ^ {(k)}), \quad k = 1, \dots , L - 1
$$
where  $\mathbf{W}^{(k - 1)}\in \mathbb{R}^{n_k\times n_{k - 1}}$  is the weight matrix in the  $k^{\mathrm{th}}$  hidden layer, and  $\sigma (\cdot)$  is a nonlinear activation function applied element-wise. We consider the case of a multi-label classification, i.e. the output layer is a vector
$$
\mathbf {z} := \mathbf {z} ^ {(L)} \in \mathbb {R} ^ {d},
$$
which is fed into the  $\alpha$ -ReLU loss:
$$
\ell (\mathbf {z}, y) = \left(\alpha - \operatorname {R e L U} (\mathbf {z}) - \mathbf {e} _ {y}\right) ^ {\top} \left(\mathbf {z} - \frac {\tau}{\alpha - 1} \mathbf {1}\right) + \mathrm {H} _ {\alpha} [ \alpha - \operatorname {R e L U} (\mathbf {z}) ], \tag {15}
$$
where  $\mathrm{H}_{\alpha}[\mathbf{p}]\coloneqq \frac{1}{\alpha(\alpha - 1)}\sum_{j}(p_{j} - p_{j}^{\alpha}),\alpha \neq 1,$  is the Tsallis  $\alpha$  entropy (Tsallis, 1988). Given a training sample  $S\coloneqq \{(x,y)\}$  learning is performed by minimizing the training error
$$
\mathcal {L} := \mathbb {E} _ {(x, y) \sim S} [ \ell (\mathbf {z} (x), y) ] \tag {16}
$$
with respect to the network parameters  $\pmb{\theta} \coloneqq \operatorname{vec}\left(\{\mathbf{W}^{(k-1)}\}_{k \in [L-1]}\right)$ .
Lemma 4. Let the training error (16) be minimized by gradient descent with infinitesimally small learning rate. Let  $\mathbf{z}(x,t)\in \mathbb{R}^d$  be the network output on any training instance  $x$  at time  $t$ , and  $y$  be the desired output. Then, as the widths of hidden layers  $n_k\to \infty$ ,  $\forall k\in [L - 1]$ , the output  $\mathbf{z}(x,t)$  follows the following evolution
$$
\frac {d \mathbf {z}}{d t} = - \underset {\left(x ^ {\prime}, y ^ {\prime}\right) \sim S} {\mathbb {E}} \left[ \mathbf {K} \left(x, x ^ {\prime}\right) \cdot \left(\alpha - \operatorname {R e L U} \left(\mathbf {z} ^ {\prime}\right) - \mathbf {e} _ {y ^ {\prime}}\right) \right], \tag {17}
$$
where  $\mathbf{K}(x,x^{\prime})\in \mathbb{R}^{d\times d}$  is a positive semidefinite matrix, and  $\mathbf{z}'\coloneqq \mathbf{z}(x',t)$ .
Proof. From (16) and Lemma 1 we have
$$
\nabla_ {\mathbf {z}} \mathcal {L} = \nabla_ {\mathbf {z}} \mathbb {E} _ {(x ^ {\prime}, y ^ {\prime}) \sim S} [ \ell (\mathbf {z} ^ {\prime}, y) ] = \nabla_ {\mathbf {z}} \ell (\mathbf {z}, y) = \alpha - \operatorname {R e L U} (\mathbf {z}) - \mathbf {e} _ {y}, \tag {18}
$$
where we denoted  $\mathbf{z} \coloneqq \mathbf{z}(x, t)$  and  $\mathbf{z}' \coloneqq \mathbf{z}(x', t)$  for shorthand. Now, consider the gradient descent update
$$
\boldsymbol {\theta} _ {t + \eta} = \boldsymbol {\theta} _ {t} - \eta \nabla_ {\boldsymbol {\theta}} \mathcal {L} \quad \Leftrightarrow \quad \frac {\boldsymbol {\theta} _ {t + \eta} - \boldsymbol {\theta} _ {t}}{\eta} = - \nabla_ {\boldsymbol {\theta}} \mathcal {L}, \tag {19}
$$
where  $\eta$  is the learning rate. Taking the limit in (19) as  $\eta \to 0$ , we have:
$$
\frac {d \boldsymbol {\theta}}{d t} = - \nabla_ {\boldsymbol {\theta}} \mathcal {L} = - \mathbb {E} _ {(x ^ {\prime}, y ^ {\prime}) \sim S} [ \mathbf {J} _ {\mathbf {z} ^ {\prime}} ^ {\top} (\boldsymbol {\theta}) \cdot \nabla_ {\mathbf {z} ^ {\prime}} \mathcal {L} ],
$$
where the last equality is due to the chain rule. Combining this with (18), we get
$$
\frac {d \boldsymbol {\theta}}{d t} = - \mathbb {E} _ {\left(x ^ {\prime}, y ^ {\prime}\right) \sim S} \left[ \mathbf {J} _ {\mathbf {z} ^ {\prime}} ^ {\top} (\boldsymbol {\theta}) \cdot \left(\alpha - \operatorname {R e L U} \left(\mathbf {z} ^ {\prime}\right) - \mathbf {e} _ {y ^ {\prime}}\right) \right] \tag {20}
$$
Applying the chain rule again, and using (20), we have
$$
\frac {d \mathbf {z}}{d t} = \mathbf {J} _ {\mathbf {z}} (\boldsymbol {\theta}) \cdot \frac {d \boldsymbol {\theta}}{d t} = - \mathbb {E} _ {(x ^ {\prime}, y ^ {\prime}) \sim S} [ \underbrace {\mathbf {J} _ {\mathbf {z}} (\boldsymbol {\theta}) \mathbf {J} _ {\mathbf {z} ^ {\prime}} ^ {\top} (\boldsymbol {\theta})} _ {\mathbf {K} (x, x ^ {\prime}; \boldsymbol {\theta})} \cdot (\alpha \text {- R e L U} (\mathbf {z} ^ {\prime}) - \mathbf {e} _ {y ^ {\prime}}) ].
$$
The quantity  $\mathbf{K}(x,x^{\prime};\pmb {\theta})$  was named the Neural Tangent Kernel by Jacot et al. (2018). They also showed (see their Theorem 1) that
$$
\mathbf {K} (x, x ^ {\prime}; \boldsymbol {\theta}) \rightarrow \mathbf {K} (x, x ^ {\prime}) \quad \text {a s} \quad n _ {1}, \dots , n _ {L - 1} \rightarrow \infty ,
$$
where  $\mathbf{K}(x,x^{\prime})\in \mathbf{R}^{d\times d}$  is the deterministic kernel that does not depend on  $\theta$ . This concludes the proof.
C Instructions for Human Annotators
You are shown a reference sentence and several candidate translations. Please indicate, for each, on a 4-point scale, how much of the meaning is represented in the translation, ignoring the language quality.
Imagine you are a forgiving reader, ignoring any error that does not prevent you from getting the meaning of the text. So please ignore language oddities, typographic errors and the like. (This is difficult but key to us!)
The scale of meaning preservation is:  $4 =$  Everything /  $3 =$  Most /  $2 =$  Little /  $1 =$  None
As we are interested in comparing system's output, you can refine your judgement using  $+$  or  $-$ , e.g.  $3+$ .
When you do not know, simply leave empty.
For instance, given the reference sentence
"This restaurant is beautiful and the staff is very friendly",
valid judgements for different translations are provided in Table 6.
Score	Sentence
4	“This restaurant is beautiful and the staff is very friendly.”
4	“This restaurant is beautiful and the staff is very friendly..”
4	“Beautiful restaurant, staff is very friendly.”
4-	“This restaurant is beautiful and the staff is friendly.”
4-	“Beautiful restaurant, staff is friendly.”
2+	“Friendly staff”
2	“This is a restaurant.”
1	“Hello guys!”
1	“Bad restaurant”
1-	“Bad restaurant, bad staff”
Table 6: Evaluation example
We insist that evaluating by meaning differs from a natural intuitive evaluation. Provided the meaning is not impacted, we want to ignore the language quality, the punctuation, the casing.
D Translation Examples
Source
1.5-entmax	Flake was the central figure in Friday's drama.
Flëjk byl central'noj figuraoj v drame pjatnadcatogo
veka.	Flake was the central figure in fifteenth century
drama.
Flake was the central figure in Friday's drama.
1.5-ReLU	Flëjk byl central'noj figuraoj v drame pjatnicy.
Source	There were smiles and blue skies on Saturday (September 29) as the leaders of Turkey and Germany
met for breakfast in Berlin.
1.5-entmax	V subbotu (29 sentjabrja) byli ulbki i goluboe nebo,
poskol'ku lidery Turcii i Germanii vstretilis' dlja
razvala v Berline.	There were smiles and blue skies on Saturday
(September 29) as the leaders of Turkey and Ger-
many met for a breakup in Berlin.
1.5-ReLU	V subbotu (29 sentjabrja) byli ulbki i goluboe nebo,
tak kak lidery Turcii i Germanii vstretilis' dlja ot-
dyha v Berline.	There were smiles and blue skies on Saturday
(September 29) as the leaders of Turkey and Ger-
many met for a holiday in Berlin.
Source	That Was Really Bad Body Language:
1.5-entmax	Éto byl deolestvitel'no plohoj jazyk tela	That was really bad body language.
1.5-ReLU	Éto byl real'nyj jazyk tela	That was real body language.
Source	The city of Palu, which has more than 380,000 people,
was strewn with debris from collapsed buildings
1.5-entmax	Gorod Palu, v kotorom prozhivaet bolee 380 000
chelovek, byl razrushen zdanijami.	The city of Palu, home to over 380,000 people, was
destroyed by buildings.
1.5-ReLU	Gorod Palu, u kotorogo bolee 380 000 chelovek,
nahodilsja v updade zdanija.	The city of Palu, which has over 380,000 inhabitants, was in decay building.
Table 7: Translation Examples