SlowGuess/ACL_2020_OCR huggingface dataset:
3https://github.com/attardi/wikiextractor 4https://github.com/n-waves/multifit
7294
modern candidates for frequent historical words. Using the smallest supervised development set (500 tokens)—required by all our experiments—we compute coverage over all languages, and set the log base for squeezing the frequency ratio to 200. Finally, a penalty for rare modern forms based on their frequency is added. The score for reranking the candidate list is:  $s_{h,m} = \operatorname{ED}(h,m) + \max(\log_{200}(f_{h,m}), 0) + \frac{1}{\#(m)}$ .
Neural channel training. In every M-step (the maximization of Eq. 4), we start training from the previous EM iteration's best parameters  $\theta^{(t - 1)}$  and train for 25 epochs with 15 epochs of patience. We optimize the parameters with AdaDelta (Zeiler, 2012) using mini-batches of size 20. We do not update on candidates whose posterior probability is below  $\epsilon = 0.01\cdot \lambda^{-1}$ . If the development set scores  $\sum_{(\mathbf{x},\mathbf{y})\in \mathrm{dev}}p_{\theta}(\mathbf{x}|\mathbf{y})$  do not increase compared to the previous EM iteration, we restart training from randomly initialized parameters and using the type-level posterior probability from the best generative model found so far. That model, decoded using MAP decoding, has so far produced the highest normalization accuracy on the development set. In the semi-supervised scenario, we initially pretrain the channel for 50 epochs and 15 epochs of patience using mini-batches of size 1, as suggested by Makarov and Clematide (2018).
Sentence-wise reranker. Table 7 shows the features used in the sentence-wise PRO reranker (Hopkins and May, 2011). We learn the reranker parameters on the development set using perceptron as our binary classification learning algorithm (we also experimented with different losses and a stochastic gradient learner from the sklearn library (Pedregosa et al., 2011), but this did not produce any gains).
Direct model training. We optimize direct models with AdaDelta using mini-batches of size 10. We train for 60 epochs with 15 epochs of patience. We decode them with beam search with beam width eight. We learn a PRO reranker on the development set using hypotheses from the beam. To represent hypotheses, we use features such as the direct model probability of the hypothesis  $q_{\phi}(\hat{\mathbf{y}} \mid \mathbf{x})$ , its channel model probability  $p_{\theta}(\mathbf{x} \mid \hat{\mathbf{y}})$ , its unigram probability, the relative frequency of the (historical word, hypothesis) pair in the training data, or the edit distance between the hypothesis and the historical input word (Ta
pWORD-RNN-LM(ŷ1:m)/m
pCHAR-RNN-LM(ŷ1:m)/m
pWORD-TRIGRAM-LM(ŷ1:m)/m
length m
p(x1:m,ŷ1:m)/m
1/m ∑i=1m ED(xi,ŷi)
1 - #OOV(ŷ1:m)/m
1/m ∑i=1m pTRAIN(xi,ŷi)
1/m ∑i=1m ∑i=1n I{(xi,ŷi) ∈ TRAIN}
1/m ∑i=1m ∑i=1n I{same-suffixk(xi,ŷi)}
1/m ∑i=1m ∑i=1n I{same-prefixk(xi,ŷi)}
ble 8). We rank hypotheses with a combination of normalized edit distance (NED) and accuracy:
$$
\Delta (\mathbf {y}, \hat {\mathbf {y}}) = \mathbb {1} \left\{\mathbf {y} = \hat {\mathbf {y}} \right\} - \operatorname {N E D} (\mathbf {y}, \hat {\mathbf {y}}) \tag {7}
$$
Thus, a hypothesis  $\hat{\mathbf{y}}$  attains the highest score of  $+1$  if it is identical to the target  $\mathbf{y}$  of a development set sample and the lowest score of  $-1$  if the number of edits from  $\hat{\mathbf{y}}$  to  $\mathbf{y}$  equals the maximum of their lengths.
Table 7: Features for sentence reranker.  $\mathbf{x}_{1:m}$  is a nonstandard sentence and  $\hat{\mathbf{y}}_{1:m}$  is a standardized sentence candidate.
pUNIGRAM-LM(Ŷ)	pCHAR-RNN-LM(Ŷ)
pφ(x |Ŷ)	qφ(Ŷ | x)
NED(x,Ŷ)	ED(x,Ŷ)
pTRAIN(x,Ŷ)	(x,Ŷ) ∈ TRAIN?
same-suffixk(x,Ŷ)?	subsequence(x,Ŷ)?
same-prefixk(x,Ŷ)?	subsequence(Ŷ, x)?
Table 8: Features used to rerank hypotheses generated from the direct model.  $\mathbf{x}$  is a historical word and  $\hat{\mathbf{y}}$  is a modern language hypothesis.
7295