SlowGuess/ACL_2020_OCR huggingface dataset:
Additionally, we similarly measure the runtimes in a GPU-augmented environment (using GeForce GTX 1080 Ti). Figure 6 shows the average runtimes of the biLM and the T-TA for the number of words in a sentence. In our 20-word standard in the STS task, the T-TA takes approximately 2.51 ms, whereas biLM takes approximately 4.72 ms, showing that the T-TA is 1.88 times faster than the biLM. Compared to the CPU-only environment, the speed difference is significantly reduced due to the support offered by the GPU. Considering Figure 4, however, the CPU-only environment and GPU-augmented environment show a similar tendency: the longer the sentence is, the more significant the difference in the runtime between the T-TA and the biLM.
B.3 Perplexity and Reranking
In general, perplexity (PPL) is a measure of how well the language model is trained. To investigate the alignment of the PPL and reranking, we compute the PPL of reference sentences from the LibriSpeech dev-clean and test-clean sets using each language model. We can obtain the pseudoperplexity (pPPL) from the biLM and T-TA since they do not follow the product rule, unlike the uniLM. Note that we compute the subword-level (p)PPL (not word-level); these values are valid only in our vocabulary.
Method [WER]	(p)PPLa	(p)PPLm
dev clean	uniLM [3.82]	341.5	70.80
biLM [3.73]	(76.49)	(11.93)
T-TA [3.67]	(293.4)	(11.69)
test clean	uniLM [4.05]	495.5	73.18
biLM [3.97]	(75.43)	(12.72)
T-TA [3.97]	(590.0)	(12.43)
Table 7: (pseudo)Perplexities and corresponding WERs of the language models on LibriSpeech.
We find that the WERs are better aligned with the median of  $\mathrm{pPPL_m}$  than with the average  $\mathrm{pPPL_a}$ . Interestingly, the  $\mathrm{pPPL_a}$  of the T-TA is similar to the  $\mathrm{PPL_a}$  of the uniLM, but the  $\mathrm{pPPL_m}$  of the T-TA is similar to that of the biLM. We additionally discover that if the length of a sentence is short, the T-TA shows a very high PPL, even higher than that of the uniLM.