SlowGuess/ACL_2020_OCR huggingface dataset:
To evaluate the language models, we conduct experiments on the unsupervised task of reranking the  $N$ -best list. In these experiments, we apply each language model to rerank the 50 best candidate sentences, which are obtained in advance using each sequence-to-sequence model on ASR and NMT. The ASR and NMT models we implement are detailed in Appendices A.1 and A.2.
We rescore the sentences by linearly interpolating two scores from a sequence-to-sequence model and each language model as follows:
$$
\operatorname {s c o r e} = (1 - \lambda) \cdot \operatorname {s c o r e} _ {s 2 s} + \lambda \cdot \operatorname {s c o r e} _ {l m},
$$
where  $\mathrm{score}_{s2s}$  is the score from the sequence-to-sequence model,  $\mathrm{score}_{lm}$  is the score from the language model calculated by the sum (or mean) of the log-likelihood of each token, and the interpolation weight  $\lambda$  is set to a value that leads to the best performance in the development set.
One of the strong baseline language models, the pretrained BERT-base-uncased model (Devlin et al., 2019), is used for reranking tasks. We also include the reranking results from the traditional count-based 5-gram language models trained on each dataset using the KenLM library (Heafield, 2011).
We note that the T-TA and biLM (including BERT) assign the pseudo-log-likelihood to the score of a given sentence, whereas the uniLM assigns the log-likelihood. Because the reranking task is based on the relative scores of the  $n$ -best hypotheses, the fact that the bidirectional models yields the pseudo-log-likelihood of a given sentence does not impact this task (Wang and Cho, 2019; Shin et al., 2019; Salazar et al., 2019).
5.3.1 Results on ASR
For reranking in ASR, we use prepared  $N$ -best lists obtained from dev and test sets using Seq2SeqASR, which we train on the LibriSpeech ASR corpus. Additionally, we use the  $N$ -best lists obtained from (Shin et al., 2019) to confirm the robustness of the language models in a testing environment. Table 1 shows the word error rates (WERs) for each method after reranking. The interpolation weights  $\lambda$  are 0.3 or 0.4 in all  $N$ -best lists for ASR.
First, we confirm that the bidirectional models trained with the LAE (T-TA) and MLM (biLM) objectives consistently outperform the uniLM trained with the CLM objective. The performance gains from reranking are much lower in the better base system Seq2SeqASR, and it is evidently challenging to rerank the  $N$ -best list using a language model if the speech recognition model performs well enough. Interestingly, the T-TA is competitive with (or even better than) the biLM; this may result from the gap between the training and testing of the biLM: the biLM predicts multiple masks at a time when training but predicts only one mask at a time when testing. Moreover, the 3-layer T-TA is better than the 12-layer BERT-base, showing that in-domain data are critical to language model applications.
Finally, we note that feeding an intact input to BERT (the corresponding model is denoted as "w/ BERT $\mathbf{\Lambda}_{\mathbf{M}}$ ") in Table 1) causes the model to underperform relative to the other models, demonstrating that the mask-and-predict approach is necessary for effective reranking.
Method	dev	test
clean	other	clean	other
Shin et al.	7.17	19.79	7.25	20.37
w/ n-gram	5.62	16.85	5.75	17.72
w/ *uniSANLMw	6.05	17.32	6.11	18.13
w/ *biSANLMw	5.52	16.61	5.65	17.37
w/ BERT	5.24	16.56	5.38	17.46
w/ BERT\M	7.08	19.61	7.14	20.18
w/ uniLM	5.07	16.20	5.14	17.00
w/ biLM	4.94	16.09	5.14	16.81
w/ T-TA	4.98	16.09	5.11	16.91
Seq2SeqASR	4.11	12.31	4.31	13.14
w/ n-gram	3.94	11.93	4.15	12.89
w/ BERT	3.72	11.59	3.97	12.46
w/ BERT\M	4.09	12.26	4.28	13.15
w/ uniLM	3.82	11.73	4.05	12.63
w/ biLM	3.73	11.53	3.97	12.41
w/ T-TA	3.67	11.56	3.97	12.38
Table 1: WERs after reranking with each language model on LibriSpeech. The 'other' sets are recorded in noisier environments than the 'clean' sets. Bold font denotes the best performance on each subtask, and * signifies a word-level language model from Shin et al. (2019).
5.3.2 Results on NMT
To compare the reranking performances in another domain, NMT, we again prepare  $N$ -best lists using Seq2Seq $NMT^3$  from the WMT13 German-to-English (De→En) and French-to-English (Fr→En) test sets. Table 2 shows the bilingual evaluation understudy (BLEU) scores for each method after reranking. Each interpolation weight becomes a value that shows the best performance on each test set with each method in NMT. The interpolation weights  $\lambda$  are 0.4 or 0.5 in the  $N$ -best lists for NMT.
We confirm again that the bidirectional models trained with the LAE and MLM objectives perform better than the uniLM trained with the CLM objective. Additionally, the  $\mathrm{Fr} \rightarrow \mathrm{En}$  translation has less effect on the reranking than the  $\mathrm{De} \rightarrow \mathrm{En}$  translation because the base NMT system for  $\mathrm{Fr} \rightarrow \mathrm{En}$  is better than that for  $\mathrm{De} \rightarrow \mathrm{En}$ . The 12-layer BERT model appears much better than the other models at reranking on NMT; hence, the  $N$ -best hypotheses of the NMT model seem to be more indistinguish-
3The Seq2Seq models for De→En and Fr→En are trained independently using the t2t library (Vaswani et al., 2018).
Method	De→En	Fr→En
Seq2SeqNMT	27.83	29.63
w/ n-gram	28.41	30.04
w/ BERT	29.31	30.52
w/ uniLM	28.80	30.21
w/ biLM	28.76	30.32
w/ T-TA	28.83	30.20
able than those of the ASR model from a language modeling perspective.
All the reranking results on the ASR and NMT tasks demonstrate that the proposed T-TA performs both efficiently (similar to the uniLM) and effectively (similar to the biLM).
5.4 Unsupervised STS
In addition to the reranking task, we apply the language models to an STS task, that is, measuring the similarity between the meaning of sentence pairs. We use the STS Benchmark (STS-B) (Cer et al., 2017) and Sentences Involving Compositional Knowledge (SICK) (Marelli et al., 2014) datasets, both of which have a set of sentence pairs with corresponding similarity scores. The evaluation metric of STS is Pearson's  $r$  between the predicted similarity scores and the reference scores of the given sentence pairs.
In this section, we address the unsupervised STS task to examine the inherent ability of each language model to obtain contextual language representations, and we mainly compare the language models that are trained on the English Wikipedia dump. To compute the similarity score of a given sentence pair, we use the cosine similarity of two sentence representations, where each representation is obtained by averaging each language model's contextual representations. Specifically, the contextual representations of a given sentence are the outputs of the final encoding layer of each model, denoted as context in Tables 3 and 4. For comparison, we use noncontextual representations, which are obtained from the outputs of the embedding layer, denoted as embed in Tables 3 and 4.
As a strong baseline for unsupervised STS tasks, we also include the 12-layer BERT model (Devlin
Table 2: BLEU scores after reranking with each language model on WMT13. Bold font denotes the best performance on each subtask, and the underlined values signify the best performances in our implementations.
Method	STS-B-dev	STS-B-test
context	embed	context	embed
BERT	64.78	-	54.22	-
BERT\M	59.17	60.07	47.91	48.19
BERT[CLS]	29.16	17.18
uniLM	56.25	63.87	39.57	55.00
uniLM[EOS]	40.75	38.30
biLM	59.99	-	50.76	-
biLM\M	53.20	58.80	36.51	49.08
T-TA	71.88	54.75	62.27	44.74
GloVe	-	52.4	-	40.6
Word2Vec	-	70.0	-	56.5
Table 3: Pearson's  $r \times  {100}$  results on the STS-B dataset. " -" denotes an infeasible value, and bold font denotes the top 2-performing models on each subtask.
et al., 2019), and we employ BERT in the mask-and-predict approach for computing the contextual representations of each sentence. Note that we use the most straightforward approach for the unsupervised STS task to focus on comparing token-level language representations.
5.4.1 Results on STS-B
The STS-B dataset has 5749/1500/1379 sentence pairs with train/dev/test splits and corresponding scores ranging from 0 to 5. We test the language models on the STS-B-dev and STS-B-test sets using the simplest approach on the unsupervised STS task. As additional baselines, we include the results of GloVe (Pennington et al., 2014) and Word2Vec (Mikolov et al., 2013a) from the official sites of STS Benchmark<sup>4</sup>.
Table 3 shows our T-TA trained with the LAE objective best captures the semantics of a sentence over the Transformer-based language models. Remarkably, our 3-layer T-TA trained on a relatively small dataset outperforms the 12-layer BERT trained on a larger dataset (Wikipedia + BookCorpus). Furthermore, the embedding representations are trained better by the CLM objective than by the other language modeling objectives; we suppose that the uniLM depends strongly on the embedding layer due to its unidirectional context constraint.
Since the uniLM encodes all contexts in the last token, [EOS], we also use the last representation as the sentence representation; however, this approach does not outperform the average sentence
<sup>4</sup>http://ixa2.si.ehu.es/stswiki/index.php/STSbenchmark
Method	SICK-test
context	embed
BERT	64.31	-
BERT\M	61.18	64.63
uniLM	54.20	65.69
biLM	58.98	-
biLM\M	53.79	62.67
T-TA	69.49	60.77
Table 4: Pearson's  $r \times  {100}$  results on the SICK dataset. " - " denotes an infeasible value, and bold font denotes the best performance on each subtask.
representation. Similarly, BERT has a special token, [CLS], which is trained for the "next sentence prediction" objective; thus, we also use the [CLS] token to see how this model learns the sentence representation, but it significantly underperforms the other models.
5.4.2 Results on SICK
We further evaluate the language models on the SICK dataset, which consists of 4934/4906 sentence pairs with training/testing splits and scores ranging from 1 to 5. The results are in Table 4, from which we obtain the same observations as those reported for STS-B.
All results on unsupervised STS tasks demonstrate that the T-TA learns textual semantics best using the token-level LAE objective.
In this work, we propose a novel deep bidirectional language model, namely, the T-TA, to eliminate the computational overload of applying BERT to unsupervised applications. Experimental results on  $N$ -best list reranking and unsupervised STS tasks demonstrate that the proposed T-TA is significantly faster than the BERT-like approach, and its encoding ability is competitive with (or even better than) that of BERT.
K. Jung is with ASRI, Seoul National University, Korea. This work was supported by the Ministry of Trade, Industry & Energy (MOTIE, Korea) under the Industrial Technology Innovation Program (No.10073144) and by the NRF grant funded by the Korean government (MSIT) (NRF2016M3C4A7952587).
Ebru Arisoy, Abhinav Sethy, Bhuvana Ramabhadran, and Stanley Chen. 2015. Bidirectional recurrent neural network language models for automatic speech recognition. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5421-5425. IEEE.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of machine learning research, 3(Feb):1137-1155.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1-14.
William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. 2016. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4960-4964. IEEE.
Xie Chen, Anton Ragni, Xunying Liu, and Mark JF Gales. 2017. Investigating bidirectional recurrent neural network language models for speech recognition. In INTERSPEECH, pages 269-273.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724-1734.
Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. 2015. Attention-based models for speech recognition. In Advances in neural information processing systems, pages 577-585.
Alexis Conneau and Guillaume Lample. 2019. Crosslingual language model pretraining. In Advances in Neural Information Processing Systems, pages 7057-7067.
Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In Advances in neural information processing systems, pages 3079-3087.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
William H DuBay. 2006. The classic readability studies. Impact Information, Costa Mesa, California.
Kenneth Heafield. 2011. Kenlm: Faster and smaller language model queries. In Proceedings of the sixth workshop on statistical machine translation, pages 187-197. Association for Computational Linguistics.
Dan Hendrycks and Kevin Gimpel. 2016. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. arXiv preprint arXiv:1606.08415.
Takaaki Hori, Shinji Watanabe, Yu Zhang, and William Chan. 2017. Advances in joint ctc-attention based end-to-end speech recognition with a deep cnn encoder and rnN-lm. Proc. Interspeech 2017, pages 949-953.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations.
Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014. Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014), pages 1-8.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Eleventh annual conference of the international speech communication association.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206-5210. IEEE.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference
on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Alvaro Peris and Francisco Casacuberta. 2015. A bidirectional recurrent neural language model for machine translation. Procesamento del Lenguaje Natural, 55:109-116.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 2227-2237.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392.
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. 1986. Learning representations by backpropagating errors. nature, 323(6088):533-536.
Julian Salazar, Davis Liang, Toan Q Nguyen, and Katrin Kirchhoff. 2019. Pseudolikelihood reranking with masked language models. arXiv preprint arXiv:1910.14659.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96.
Yusuxke Shibata, Takuya Kida, Shuichi Fukamachi, Masayuki Takeda, Ayumi Shinohara, Takeshi Shinohara, and Setsuo Arikawa. 1999. Byte pair encoding: A text compression scheme that accelerates pattern matching. Technical report, Technical Report DOI-TR-161, Department of Informatics, Kyushu University.
Joonbo Shin, Yoonhyung Lee, and Kyomin Jung. 2019. Effective sentence scoring method using bert for speech recognition. In *Asian Conference on Machine Learning*, pages 1081-1093.
Martin Sundermeyer, Ralf Schluter, and Hermann Ney. 2012. LSTM neural networks for language modeling. In Thirteenth annual conference of the international speech communication association.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104-3112.
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, et al. 2018. Tensor2tensor for neural machine translation. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Papers), pages 193-199.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008.
Alex Wang and Kyunghyun Cho. 2019. Bert has a mouth, and it must speak: Bert as a markov random field language model. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 30-36.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019.
Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson-Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, et al. 2018. Espnet: End-to-end speech processing toolkit. Proc. Interspeech 2018, pages 2207-2211.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
Matthew D Zeiler. 2012. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701.
A.1 Setup for the ASR System
This section introduces our implementation of the ASR system.
For the input features, we use an 80-band Mel-scale spectrogram derived from the speech signal. The target sequence is processed in  $5\mathrm{K}$  case-insensitive subword units created via unigram byte-pair encoding (Shibata et al., 1999). We use an attention-based encoder-decoder model as our acoustic model. The encoder is a 5-layer bidirectional long short-term memory (LSTM) network, and there are bottleneck layers that conduct a linear transformation between every LSTM layer. Additionally, there is a VGG module before the encoder, and it reduces the number of encoding time steps by one-quarter through two max-pooling layers. The decoder is a 2-layer bidirectional LSTM network with a location-aware attention mechanism (Chorowski et al., 2015). All the layers have 1024 hidden units. The model is trained with an additional connectionist temporal classification (CTC) objective function because the left-to-right constraint of CTC helps learn alignments between speech-text pairs (Hori et al., 2017).
Our model is trained for 20 epochs on  $960\mathrm{h}$  of LibriSpeech training data using the Adadelta optimizer (Zeiler, 2012). Using this acoustic model, we obtain the 50 best decoded sentences for each input audio file through the hybrid CTC-attention-based scoring (Hori et al., 2017) method. For Seq2SeqASR, we additionally use a pretrained recurrent neural network language model (RNNLM) to combine the log-probability  $p^{lm}$  of the RNNLM during decoding as follows:
$$
\begin{array}{l} \log p \left(y _ {n} \mid y _ {1: n - 1}\right) \\ = \log p ^ {\mathrm {a m}} \left(y _ {n} \mid y _ {1: n - 1}\right) + \beta \log p ^ {\mathrm {l m}} \left(y _ {n} \mid y _ {1: n - 1}\right), \tag {3} \\ \end{array}
$$
where  $\beta$  is set to 0.7. We use the efficient spatial pyramid network (ESPNet) toolkit (Watanabe et al., 2018) for this implementation.
Table 5 shows the oracle word error rates (WERs) of the 50 best lists measured assuming that the best sentence is always picked from the candidates. We also include the oracle WERs from the 50 best lists of (Shin et al., 2019).
Method	dev	test
clean	other	clean	other
Shin et al.	7.17	19.79	7.26	20.37
oracle	3.18	12.98	3.19	13.61
Seq2SeqASR	4.11	12.31	4.31	13.14
oracle	1.80	7.90	1.96	8.39
A.2 Setup for the NMT System
We implement the standard Transformer model (Vaswani et al., 2017) using the Tensor2Tensor library (Vaswani et al., 2018) for NMT. Both the encoder and the decoder of the Transformer consist of 6 layers with 512 hidden units, and the number of self-attention heads is 8. The maximum number of input tokens is set to 256, and we use a shared vocabulary of size  $32k$ . For effective training, we let the token embedding layer and the last softmax layer share their weights. The other hyperparameters of our translation system follow the standard transformer_base_singlegpu setting in Google's official Tensor2Tensor repository<sup>5</sup>.
We train the baseline model on the standard WMT13 Fr→En and De→En datasets with 250k steps using the Adam optimizer (Kingma and Ba, 2014). We use linear-warmup-square-root-decay learning rate scheduling with the default learning rate (2.5e-4) and number of warmup steps (16k). Using this baseline translation model, we obtain the 50 best decoded sentences for each source through the beam search. The oracle BLEU scores for the NMT system are shown in Table 6.
Table 5: Oracle WERs of the 50 best lists on LibriSpeech from each ASR system.
Method	WMT13
De→En	Fr→En
Seq2SeqNMT	27.83	29.63
oracle	38.18	39.58
Table 6: Oracle BLEU scores of the 50 best lists on WMT13
B.1 Runtimes of the uniLM and T-TA
As mentioned in Section 5.2, we also measure the runtimes of the uniLM we implement. Figure 5
$^{5}$ https://github.com/tensorflow/tensor2tensor
Figure 5: Runtimes according to the number of words for the uniLM and T-TA.
Figure 6: Runtimes according to the number of words for the biLM and T-TA in the GPU-augmented environment.
shows the average runtimes of the uniLM and the T-TA for the number of words in a sentence. Since we use subword tokens, the number of words  $n_w$  and the number of tokens  $n$  can be different ( $n_w \leq n$ ).
B.2 Runtimes on a GPU