SlowGuess/ACL_2020_OCR huggingface dataset:
Joongbo Shin, Yoonhyung Lee, Seunghyun Yoon, Kyomin Jung
{jbshin, cpi1234, mysmilish, kjung} @snu.ac.kr
Even though BERT has achieved successful performance improvements in various supervised learning tasks, BERT is still limited by repetitive inferences on unsupervised tasks for the computation of contextual language representations. To resolve this limitation, we propose a novel deep bidirectional language model called a Transformer-based Text Autoencoder (T-TA). The T-TA computes contextual language representations without repetition and displays the benefits of a deep bidirectional architecture, such as that of BERT. In computation time experiments in a CPU environment, the proposed T-TA performs over six times faster than the BERT-like model on a reranking task and twelve times faster on a semantic similarity task. Furthermore, the T-TA shows competitive or even better accuracies than those of BERT on the above tasks. Code is available at https://github.com/joongbo/ta.
A language model is an essential component of many natural language processing (NLP) applications ranging from automatic speech recognition (ASR) (Chan et al., 2016; Panayotov et al., 2015) to neural machine translation (NMT) (Sutskever et al., 2014; Sennrich et al., 2016; Vaswani et al., 2017). Recently, the Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2019) and its variations have led to significant improvements in learning natural language representation and have achieved state-of-the-art performances on various downstream tasks such as the General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2019) and question answering (Rajpurkar et al., 2016). BERT continues to succeed in various unsupervised tasks, such as the  $N$ -best list reranking for ASR and NMT (Shin et al., 2019; Salazar et al., 2019), con-
firming that deep bidirectional language models are useful in unsupervised applications as well.
However, concerning its applications to unsupervised learning tasks, BERT is significantly inefficient at computing language representations at the inference stage (Salazar et al., 2019). During training, BERT adopts the masked language modeling (MLM) objective, which is to predict the original word of the explicitly masked word from the input sequence. Following the MLM objective, each contextual word representation should be computed by a two-step process: masking a word in the input and feeding the result to BERT. During the inference stage, this process is repeated  $n$  times to obtain the representations of all the words within a text sequence (Wang and Cho, 2019; Shin et al., 2019; Salazar et al., 2019), resulting in a computational complexity of  $O(n^{3})^{1}$  in terms of the number of words  $n$ . Hence, it is necessary to reduce the computational complexity when applying the model to situations where the inference time is critical, e.g., mobile environments and real-time systems (Sanh et al., 2019; Lan et al., 2019). Considering this limitation of BERT, we submit a new research question: "Can we construct a deep bidirectional language model with a minimal inference time while maintaining the accuracy of BERT?"
In this paper, in response to the above question, we propose a novel bidirectional language model named the Transformer-based Text Autoencoder (T-TA), which has a reduced computational complexity of  $O(n^{2})$  when applying the model to unsupervised applications. The proposed model is trained with a new learning objective named language autoencoding (LAE). The LAE objective, which allows the target labels to be the same as the text input, is to predict every token in the input sequence simultaneously without merely copying
<sup>1</sup>A complexity of  $O(n^{2})$  is derived from the per-layer complexity of the Transformer (Vaswani et al., 2017).
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 823-835 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
the input to the output. To learn the proposed objective, we devise both a diagonal masking operation and an input isolation mechanism inside the T-TA based on the Transformer encoder (Vaswani et al., 2017). These components enable the proposed T-TA to compute contextualized language representations at once while maintaining the benefits of the deep bidirectional architecture of BERT.
We conduct a series of experiments on two unsupervised tasks:  $N$ -best list reranking and unsupervised semantic textual similarity. First, by conducting runtime experiments in a CPU environment, we show that the proposed T-TA is 6.35 times faster than the BERT-like model in the reranking task and 12.7 times faster in the unsupervised semantic textual similarity task. Second, despite its faster inference time, the T-TA achieves competitive performances relative to BERT on reranking tasks. Furthermore, the T-TA outperforms BERT by up to 8 points in Pearson's  $r$  on unsupervised semantic textual similarity tasks.
When referring to an autoencoder for language modeling, sequence-to-sequence learning approaches have been commonly used. These approaches encode a given sentence into a compressed vector representation, followed by a decoder that reconstructs the original sentence from the sentence-level representation (Sutskever et al., 2014; Cho et al., 2014; Dai and Le, 2015). To the best of our knowledge, however, none of these approaches consider an autoencoder that encodes word-level representations (such as BERT) without an autoregressive decoding process.
Many studies have been performed on neural network-based language models for word-level representations. Distributed word representations were proposed and attracted considerable interest, as they were considered to be fundamental building blocks for NLP tasks (Rumelhart et al., 1986; Bengio et al., 2003; Mikolov et al., 2013b). Subsequently, researchers explored contextualized representations of text where each word has a different representation depending on the context (Peters et al., 2018; Radford et al., 2018). Most recently, a Transformer-based deep bidirectional model was proposed and applied to various supervised-learning tasks with remarkable success (Devlin et al., 2019).
For unsupervised tasks, researchers have adopted
recently developed language-representation models and investigated their effectiveness; a typical example is the  $N$ -best list reranking for ASR and NMT tasks. In particular, studies have integrated left-to-right and right-to-left language models (Arisoy et al., 2015; Chen et al., 2017; Peris and Casacuberta, 2015) to outperform conventional unidirectional language models (Mikolov et al., 2010; Sundermeyer et al., 2012) in these tasks. Furthermore, BERT-based approaches have been explored and have achieved significant performance improvements on these tasks because bidirectional language models yield the pseudo-log-likelihood of a given sentence, and this score is useful in ranking the  $n$ -best hypotheses (Wang and Cho, 2019; Shin et al., 2019; Salazar et al., 2019).
Another line of research involves reducing the computation time and memory consumption of BERT. Lan et al. (2019) proposed parameter-reduction techniques, factorized embedding parameterization and cross-layer parameter sharing and reported 18 times fewer parameters and a 1.7-fold increase in the training time. Similarly, Sanh et al. (2019) presented a method to pretrain a smaller model that can be fine-tuned for downstream tasks and achieved 1.4 times fewer parameters with a 1.6-fold increase in the inference time. However, none of these studies developed methods that directly revise the BERT architecture to reduce the computational complexity during the inference stage.
3 Language Model Baselines
In a conventional language modeling task, the  $i$ -th token  $x_{i}$  is predicted using its preceding context  $\mathbf{x}_{< i} = [x_{1},\dots,x_{i - 1}]$ ; throughout this paper, this objective is known as causal language modeling (CLM) following (Conneau and Lample, 2019). As shown in Figure 1a, we can obtain (left-to-right) contextualized language representations  $\mathbf{H}^{\mathrm{C}} = [H_1^{\mathrm{C}},\dots,H_n^{\mathrm{C}}]$  after feeding the input sequence to the CLM-trained language model only once, where  $H_{i}^{\mathrm{C}} = h^{\mathrm{C}}(\mathbf{x}_{< i})$  is the hidden representation of the  $i$ -th token. This paper takes this unidirectional language model (uniLM) as our speed baseline. However, contextualized language representations obtained from the uniLM are insufficient to accurately encode a given text because future contexts cannot be leveraged to understand the current tokens during the inference stage.
Recently, BERT (Devlin et al., 2019) was designed to enable the full contextualization
(a) Causal language modeling
(b) Masked language modeling
(c) Language autoencoding
Figure 1: Schematic diagrams of language models for the (a) CLM, (b) MLM, and (c) LAE objectives.
of language representations by using the MLM objective, in which some tokens from the input sequence are randomly masked; the objective is to predict the original tokens at the masked positions using only their context. As in Figure 1b, we can obtain a contextualized representation of the  $i$ -th token  $H_{i}^{\mathrm{M}} = h^{\mathrm{M}}(M_{i}(\mathbf{x}))$  by masking the token in the input sequence and feeding it to the MLM-trained model, where  $M_{i}(\mathbf{x}) = [x_{1}, \ldots, x_{i-1}, [\mathrm{MASK}], x_{i+1}, \ldots, x_{n}]$  signifies an external masking operation. This paper takes this bidirectional language model (biLM) as our performance baseline. However, this mask-and-predict approach should be repeated  $n$  times to obtain all the language representations  $\mathbf{H}^{\mathrm{M}} = [H_{1}^{\mathrm{M}}, \ldots, H_{n}^{\mathrm{M}}]$  because learning occurs only at the masked position during the MLM training stage. Although the resulting language representations are robust and accurate, as a consequence of this repetition, the model is significantly inefficient when applied to unsupervised tasks such as  $N$ -best list reranking (Wang and Cho, 2019; Shin et al., 2019; Salazar et al., 2019).
4.1 Language Autoencoding
In this paper, we propose a new learning objective named language autoencoding (LAE) for obtaining fully contextualized language representations without repetition. The LAE objective, with which the output is the same as the input, is to predict every token in a text sequence simultaneously without merely copying the input to the output. For the proposed task, a language model should reproduce the whole input at once while avoiding overfitting; otherwise, the model outputs only the representation copied from the input representation without learning any statistics of the language. To this end, the flow of information from the
$i$ -th input to the  $i$ -th output should be blocked inside the model shown in Figure 1c. From the LAE objective, we can obtain fully contextualized language representations  $\mathbf{H}^{\mathrm{L}} = [H_1^{\mathrm{L}}, \ldots, H_n^{\mathrm{L}}]$  all at once, where  $H_i^{\mathrm{L}} = h^{\mathrm{L}}(\mathbf{x}_{\backslash i})$  and  $\mathbf{x}_{\backslash i} = [x_1, \ldots, x_{i-1}, x_{i+1}, \ldots, x_n]$ . The method for blocking the flow of information is described in the next section.
4.2 Transformer-based Text Autoencoder
In this section, we introduce the novel architecture of the proposed T-TA shown in Figure 2. As indicated by its name, the T-TA architecture is based on the Transformer encoder (Vaswani et al., 2017). To learn the proposed LAE objective, we develop both a diagonal masking operation and an input isolation mechanism inside the T-TA. Both developments are designed to enable the language model to predict all tokens simultaneously while maintaining the deep bidirectional property (see the descriptions in the following subsections). For brevity, we refer to the original paper on the Transformer encoder (Vaswani et al., 2017) for other details regarding the standard functions, such as the multihead attention and scaled dot-product attention mechanisms, layer normalization, and the position-wise fully connected feed-forward network.
4.2.1 Diagonal Masking
As shown in Figure 3, a diagonal masking operation is implemented inside the scaled dot-product attention mechanism to be "self-unknown" during the inference stage. This operation prevents information from flowing to the same position in the next layer by masking out the diagonal values in the input of the softmax function. Specifically, the output vector at each position is the weighted sum of the value  $\mathbf{V}$  at other positions, where the attention weights come from the query  $\mathbf{Q}$  and the key  $\mathbf{K}$ .
Figure 2: Architecture of our T-TA. The highlighted box and dashed arrows are the innovations presented in this paper.
The diagonal mask becomes meaningless when we use it together with a residual connection or utilize it within the multilayer architecture. To retain the self-unknown functional, we can remove the residual connection and adopt a single-layer architecture. However, it is essential to utilize a deep architecture to understand the intricate patterns of natural language. To this end, we further develop the architecture described in the next section.
4.2.2 Input Isolation
We now propose an input isolation mechanism to ensure that the residual connection and the multi-layer architecture are compatible with the above-mentioned diagonal masking operation. In the input isolation mechanism, the key and value inputs  $(\mathbf{K}$  and  $\mathbf{V}$ , respectively) of all encoding layers are isolated from the network flow and are fixed to the sum of the token embeddings and the position embeddings. Hence, only the query inputs  $(\mathbf{Q})$  are updated across the layers during the inference stage by referring to the fixed output of the embedding layer.
Figure 3: Diagonal masking of the scaled dot-product attention mechanism. The highlighted box and dashed arrow represent the innovations reported in this paper.
Additionally, we input the position embeddings to the  $\mathbf{Q}$  of the very first encoding layer, thereby making the self-attention mechanism effective. Otherwise, the attention weights will be the same at all positions, and thus, the first self-attention mechanism will function as a simple average of all the input representations (except the "self" position). Finally, we apply the residual connection only to the query to completely maintain unawareness. The dashed arrows in Figure 2 show the proposed input isolation mechanism inside the T-TA.
By using diagonal masking and input isolation in conjunction, the T-TA can have multiple encoder layers, enabling the T-TA to obtain high-quality contextual language representations after feeding a sequence into the model only once.
4.3 Discussion and Analysis
Heretofore, we have introduced the new learning objective named LAE, and the novel deep bidirectional language model named T-TA. We will verify the architecture of the proposed T-TA in Section 4.3.1 and compare our model with the recently proposed strong baseline BERT in Section 4.3.2.
4.3.1 Verification of the Architecture
Here, we discuss how diagonal masking with input isolation preserves the "self-unknown" property in detail.
As shown in Figure 2, we have two input embeddings, namely, token embeddings  $\mathbf{X} = [X_1,\dots ,X_n]^T\in \mathbb{R}^{n\times d}$  and position embeddings  $\mathbf{P} = [P_{1},\ldots ,P_{n}]^{T}\in \mathbb{R}^{n\times d}$ , where  $d$  is an embedding dimension. From the input isolation mechanism, the key and value  $\mathbf{K} = \mathbf{V} = \mathbf{X} + \mathbf{P}$  have the information of the input tokens and are fixed in
all layers, but the query  $\mathbf{Q}^l$  is updated across the layers during the inference stage starting from the position embeddings  $\mathbf{Q}^{1} = \mathbf{P}$  in the first layer.
Let us consider the  $l$ -th encoding layer's query input  $\mathbf{Q}^l$  and its output  $\mathbf{H}^l = \mathbf{Q}^{l + 1}$ :
$$
\begin{array}{l} \mathbf {H} ^ {l} = \operatorname {S M S A N} (\mathbf {Q} ^ {l}, \mathbf {K}, \mathbf {V}) \\ = g \left(\operatorname {N o r m} \left(\operatorname {A d d} \left(\mathbf {Q} ^ {l}, f \left(\mathbf {Q} ^ {l}, \mathbf {K}, \mathbf {V}\right)\right)\right)\right), \tag {1} \\ \end{array}
$$
where  $\mathrm{SMSAN}(\cdot)$  is the self-masked self-attention network, namely, the encoding layer of the T-TA,  $g(x) = \mathrm{Norm}(\mathrm{Add}(x,\mathrm{FeedForward}(x)))$  signifies two upper subboxes of the encoding layer in Figure 2, and  $f(\cdot)$  is the (multihead) diagonal-masked self-attention (DMSA) mechanism. As illustrated in Figure 3, the DMSA module computes  $\mathbf{Z}^l$  as follows:
$$
\begin{array}{l} \mathbf {Z} ^ {l} = f (\mathbf {Q} ^ {l}, \mathbf {K}, \mathbf {V}) = \mathrm {D M S A} (\mathbf {Q} ^ {l}, \mathbf {K}, \mathbf {V}) \\ = \operatorname {S o f t M a x} (\operatorname {D i a g M a s k} \left(\mathbf {Q} ^ {l} \mathbf {K} ^ {T} / \sqrt {d}\right)) \mathbf {V}. \tag {2} \\ \end{array}
$$
In the DMSA module, the  $i$ -th element of  $\mathbf{Z}^l = [Z_1^l,\ldots ,Z_n^l ]^T$  is always computed by a weighted average of the fixed  $\mathbf{V}$  while discarding the information of the  $i$ -th token  $X_{i}$  in  $V_{i}$ . Specifically,  $Z_{i}^{l}$  is the weighted average of  $\mathbf{V}$  with the attention weight vector  $\mathbf{s}_i^l$ , i.e.,  $Z_{i}^{l} = \mathbf{s}_{i}^{l}\mathbf{V}$ , where  $\mathbf{s}_i^l = [s_1^l,\dots,s_{i - 1}^l,0,s_{i + 1}^l,\dots,s_n^l]\in \mathbb{R}^{1\times n}$ . Here, we note that the DMSA mechanism is related only to the "self-unknown" property since no token representations are referred to each other in subsequent transformations from  $\mathbf{Z}^l$  to  $\mathbf{H}^l$ . Therefore, we can guarantee that the  $i$ -th element of the query representation in any layer,  $Q_{i}^{l}$ , never encounters the corresponding token representation starting from  $Q_{i}^{1} = P_{i}$ . Consequently, the T-TA preserves the "self-unknown" property during the inference stage while maintaining the residual connection and multilayer architecture.
4.3.2 Comparison with BERT
There are several differences between the strong baseline BERT (Devlin et al., 2019) and the proposed T-TA, while both models learn deep bidirectional language representations.
- While BERT uses an external masking operation in the input, the T-TA has an internal masking operation in the model, as we intend. Additionally, while BERT is based on a denoising autoencoder, the T-TA is based on an autoencoder. With this novel approach, the T-TA does not need
mask-and-predict repetition during the computing of contextual language representations. Consequently, we reduce the computational complexity from  $O(n^{3})$  with the BERT to  $O(n^{2})$  with the T-TA in applications to unsupervised learning tasks.
- As in the T-TA, feeding an intact input (without masking) into BERT is also possible. However, we argue that this process will significantly diminish the model performance in unsupervised applications since the MLM objective does not consider intact tokens much. In the next section, we include experiments that reveal the model performance with intact inputs (described in Tables 1, 3, and 4). For further reference, we also suggest a previous study that reported the same opinion (Salazar et al., 2019).
To evaluate the proposed method, we conduct a series of experiments. We first evaluate the contextual language representations obtained from the T-TA on  $N$ -best list reranking tasks. We then apply our method to unsupervised semantic textual similarity (STS) tasks. The following sections will demonstrate that the proposed model is much faster than BERT during the inference stage (Section 5.2) while showing competitive or even better accuracies than those of BERT on reranking tasks (Section 5.3) and STS tasks (Section 5.4).
5.1 Language Model Setups
The main purpose of this paper is to compare the proposed T-TA with a biLM trained with the MLM objective. For a fair comparison, each model has the same number of parameters based on the Transformer as follows:  $|L| = 3$  self-attention layers with  $d = 512$  input and output dimensions,  $h = 8$  attention heads, and  $d_f = 2048$  hidden units for the position-wise feed-forward layers. We use a Gaussian error linear unit (gelu) activation function (Hendrycks and Gimpel, 2016) rather than the standard rectified linear unit (relu) following OpenAI GPT (Radford et al., 2018) and BERT (Devlin et al., 2019). In our experiments, we set the position embeddings to be trainable following BERT (Devlin et al., 2019) rather than a fixed sinusoid (Vaswani et al., 2017) with supported sequence lengths up to 128 tokens. We use WordPiece embeddings (Wu et al., 2016) with a vocabulary of approximately  $|V| \simeq 30,000$  tokens. The weights
of the embedding layer and the last softmax layer of the Transformer are shared. For the speed baseline, we also implement a uniLM that has the same number of parameters as the T-TA and biLM.
For training, we create a training instance consisting of a single sentence with [BOS] and [EOS] tokens at the beginning and end of each sentence, respectively. We use 64 sentences as the training batch and train the language models over  $1M$  steps for ASR and  $2M$  steps for NMT. We train the language models with Adam (Kingma and Ba, 2014) with an initial learning rate of  $1e - 4$  and coefficients of  $\beta_{1} = 0.9$  of  $\beta_{2} = 0.999$ ; the learning rate is set to warm up over the first  $50k$  steps, and the learning rate exhibits linear decay. We use a dropout probability of 0.1 on all layers. Our implementation is based on Google's official code for BERT<sup>2</sup>.
To train the language models that we implement, we use an English Wikipedia dump (approximately 13 GB in size) containing approximately  $120M$  sentences. The trained models are used for reranking in NMT and unsupervised STS tasks. For the ASR reranking task, we use additional in-domain training data, namely, 4.0 GB of normalized text data from the official LibriSpeech corpus containing approximately  $40M$  sentences.
5.2 Runtime Analysis
We first measure the runtime of each language model to compute the contextual language representation  $\mathbf{H}^L\in \mathbb{R}^{n\times d}$  of a given text sequence. In the unsupervised STS tasks, we directly use  $\mathbf{H}^L$  for the analysis. In the case of the reranking task, further computation is required: we compute Softmax  $(\mathbf{H}^L\mathbf{E}^T)$  to obtain the likelihood of each token, where  $\mathbf{E}\in \mathbb{R}^{|V|\times d}$  is the weight parameter of the softmax layer. Therefore, the computational complexity of the reranking task is larger than that of the STS task.
To measure the runtime, we use an Intel(R) Core(TM) i7-6850K CPU (3.60 GHz) and the TensorFlow 1.12.0 library with Python 3.6.8 on Ubuntu 16.04.06 LTS. In each experiment, we measure the runtime 50 times and average the results.
Figure 4 shows that the T-TA exhibits faster runtimes than the biLM, and the gap between the T-TA and biLM increases as the sentence becomes longer. To facilitate a numerical comparison, we set the standard number of words to 20, which is approxi
Figure 4: Average runtimes of each model according to the number of words on STS and reranking tasks, subscribed as  $sts$  and  $rrk$ , respectively.
mately the average number of words in a contemporary English sentence (DuBay, 2006). In this setup, in the STS tasks, the T-TA takes approximately  $9.85\mathrm{ms}$ , while the biLM takes approximately 125 ms; hence, the T-TA is 12.7 times faster than the biLM. In the reranking task, the T-TA is 6.35 times faster than the biLM (which is still significant); this reduction occurs because the repetition of the biLM is related only to computing  $\mathbf{H}^L$  rather than Softmax  $(\mathbf{H}^{L}\mathbf{E}^{T})$ .
For the visual clarity of Figure 4, we omit the runtime results of the uniLM, which is as fast as the T-TA (see Appendix B.1). With such a fast inference time, we next demonstrate that the T-TA is as accurate as BERT.
5.3 Reranking the N-best List