SlowGuess/NAACL_2022_OCR huggingface dataset:
Ziyang Luo $^{1*}$ , Yadong Xi $^{2*}$ , Jing Ma $^{1}$ , Zhiwei Yang $^{1,3}$ , Xiaoxi Mao $^{2}$ , Changjie Fan $^{2}$ , Rongsheng Zhang $^{2}$
<sup>1</sup> Department of Computer Science, Hong Kong Baptist University, Hong Kong SAR, China
$^{2}$  Fuxi AI Lab, NetEase Inc., Hangzhou, China
3 Jilin University, Jilin, China
cszyluo@comp.hkbu.edu.hk, majing@hkbu.edu.hk
{xiyadong,zhangrongsheng}@corp.netease.com
Since 2017, the Transformer-based models play critical roles in various downstream Natural Language Processing tasks. However, a common limitation of the attention mechanism utilized in Transformer Encoder is that it cannot automatically capture the information of word order, so explicit position embeddings are generally required to be fed into the target model. In contrast, Transformer Decoder with the causal attention masks is naturally sensitive to the word order. In this work, we focus on improving the position encoding ability of BERT with the causal attention masks. Furthermore, we propose a new pre-trained language model DecBERT and evaluate it on the GLUE benchmark. Experimental results show that (1) the causal attention mask is effective for BERT on the language understanding tasks; (2) our DecBERT model without position embeddings achieve comparable performance on the GLUE benchmark; and (3) our modification accelerates the pre-training process and DecBERT w/ PE achieves better overall performance than the baseline systems when pre-training with the same amount of computational resources.
In recent years, Transformer model proposed by Vaswani et al. (2017) has supplanted the widely-used LSTM (Hochreiter and Schmidhuber, 1997) as an indispensable component of many NLP systems. There are two branches of model variant: Transformer Encoder and Transformer Decoder. The Encoder-based Language Models, e.g., BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019) and DeBERTa (He et al., 2020), have achieved great success on many natural language understanding benchmarks (e.g. GLUE (Wang et al., 2019b) and SuperGLUE (Wang et al., 2019a)). The Decoder-based Language Models such as GPT-family (Radford and Narasimhan, 2018; Radford
et al., 2019; Brown et al., 2020) have shown superior performances on natural language generation. All of them utilize the Multi-Head Self-Attention (MHA) mechanism (Vaswani et al., 2017). Since MHA is designed as an order-invariant mechanism (Lee et al., 2019), Transformer Encoder without the help of position embeddings should share the same intuitions with the bag-of-word model. On the other hand, in Transformer Decoder, the causal attention masks make the MHA different from that of the Transformer Encoder. Specifically, Tsai et al. (2019) have proved that MHA with such attention masks is not permutation equivalent, indicating that Transformer Decoder is sensitive to word order.
It is noticed that several studies focus on enriching the position information of BERT to improve the performance of natural language understanding (Dai et al., 2019; Dufter et al., 2020; He et al., 2020; Wu et al., 2021a; Ke et al., 2021), e.g., introducing extra learnable parameters to trace the word order. Previous analysis also indicate that the lower layers of BERT tend to capture rich surface-level language structural information such as position information (Jawahar et al., 2019). In this paper, to improve the language understanding of BERT, we propose to enrich the position information in the lower hidden layers instead of introducing extra learnable positional parameters.
To this end, we firstly design analysis experiments to examine the effectiveness of causal attention masks in terms of capturing position information. Then we propose a new pre-trained language model DecBERT by adding the causal attention masks into the lower layers of BERT (e.g., the first two layers) to enhance the position encoding ability. In this way, our proposed model is naturally sensitive to word order. Then we pre-train our DecBERT as a masked language model, following the same objective as BERT. To verify whether our modification can help BERT trace word order, we also make a comparison with a variant of our DecBERT that
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1185 - 1197 July 10-15, 2022 ©2022 Association for Computational Linguistics
excludes any position embeddings. The experimental results show that DecBERT w/o PE has 77 times (4.59 vs. 353.97) lower valid PPL score than BERT w/o PE and achieves comparable performance with BERT w/ PE on downstream tasks, corroborating the effectiveness of our modification. Furthermore, DecBERT w/ PE achieves better performances than BERT on most downstream tasks when pre-training with the same amount of time and computational resources. By analyzing the pre-training process, we find that our modification can also accelerate pre-training.
The contributions of this work are summarised as follows:
- We propose a novel pre-train model DecBERT utilizing the causal attention masks to enhance language understanding of BERT.
- We show that DecBERT w/o PE has comparable performance with BERT w/ PE, indicating that the causal attention masks are effective for modeling word order.
- When pre-training with the same amount of time and computational resources, DecBERT  $w / PE$  achieves lower validation PPL and better overall performance on GLUE than BERT.
2 Background: Transformer
Transformer is a neural network model proposed by Vaswani et al. (2017), which relies on the multi-head self-attention (MHA) mechanism.
Input Layer. Due to the order-invariance of MHA (Lee et al., 2019), a token embedding is added with a position embedding as the input of Transformer Encoder or Decoder:
$$
h _ {i} = T E \left(x _ {i}\right) + P E (i), \tag {1}
$$
where  $x_{i}$  is a token at the  $i^{th}$  position.  $TE$  is a token embedding matrix and  $PE$  is a position embedding matrix. In the paper of Vaswani et al. (2017), they use a fixed sinusoidal  $PE$ :
$$
P E [ i, 2 j ] = \sin (i / 1 0 0 0 0 ^ {2 j / d _ {m}}), \tag {2}
$$
$$
P E [ i, 2 j + 1 ] = \cos (i / 1 0 0 0 0 ^ {2 j / d _ {m}}),
$$
where  $j$  is the dimension and  $d_{m}$  is the model size. In the later work, Devlin et al. (2019) choose to use a learnable  $PE$  matrix.
Multi-head Self-attention (MHA). MHA takes a sequence of vectors  $h = [h_1, h_2, \dots, h_n]$  as input. Then they are transformed into three different vectors, query (Q), key (K) and value (V), by three linear transformations and passed to the multi-head self-attention (MHA). The computation process of a single head is:
$$
\operatorname {A t t e n t i o n} (Q, K, V) = \operatorname {s o f t m a x} \left(\frac {Q K ^ {T}}{\sqrt {d _ {k}}}\right) V, \tag {3}
$$
where  $d_{k}$  is the dimension of a single head. MHA repeats the same process for  $h$  heads. The outputs of all heads are concatenated together and passed through a linear projection  $W^{O}$  again:
$$
\begin{array}{l} H _ {i} = \text {A t t e n t i o n} \left(Q _ {i}, K _ {i}, V _ {i}\right), \\ M H A (Q, K, V) = \operatorname {c o n c a t} \left(H _ {1}, \dots , H _ {h}\right) W ^ {O}. \tag {4} \\ \end{array}
$$
Transformer Encoder and Decoder. An Encoder layer consists of multi-head attention following with a feed-forward network (FFN). The outputs of MHA and FFN are passed through a LayerNorm (Ba et al., 2016) with residual connections (He et al., 2016). Then we stack multi-layer to form a Transformer Encoder. The difference between Decoder and Encoder is that Decoder uses the causal attention masks to mask the attention values of the subsequent tokens so that Decoder can only decode tokens relying on the tokens in the past. $^{1}$
In this section, we first analyze the relationship between Transformer Decoder and position embeddings (section 3.1). Based on this analysis, we inject the causal attention masks into BERT to create our new pre-trained language models, DecBERT (section 3.2).
3.1 Transformer Decoder and Position Embeddings
Previous studies (Tsai et al., 2019) indicate that Transformer Decoder with causal attention masks is sensitive to word order. We wonder whether Transformer Decoder can perform well without position embeddings. We assume that if Transformer Decoder without any position
<sup>1</sup>We do not consider the Encoder-Decoder Seq2seq structure with cross attention here. Encoder and Decoder are used independently.
Figure 1: Model structures of BERT and DecBERT. TRM refers to the Transformer layer.
embeddings still retains comparable performance with its counterpart with position embeddings, it will corroborate that the causal attention masks are helpful for Transformer to encode word order. To this end, we design a straightforward experiment of causal language modeling respectively on English and Chinese data as followed.
Basic Model. Our basic model is an 8-layer Transformer Decoder with 768 embedding size, 3072 feedforward layer hidden size, 12 attention heads and GELU activation function (Hendrycks and Gimpel, 2020), which is a smaller version of GPT and has 95M trainable parameters for English model and 77.5M for Chinese model. We find that if we use a standard 12-layer GPT, the number of trainable parameters will be higher than the number of tokens in the WikiText-103 dataset. This has a risk to cause over-fitting, so we choose to use an 8-layer model.
Data and Training. We resort to two publicly available wikipedia datasets. The first one is the English WikiText-103 (Merit et al., 2017). We train and evaluate our language models on the standard splits of the WikiText-103, which contains 1.8M sentences for training and 3.76k sentences for evaluation. The second one is the Chinese Wikipedia which contains about 9.28M sentences. We randomly select 34k sentences for evaluation and 9.25M for training. We use Fairseq (Ott et al., 2019) to pre-process all the data into the binary files. All the English data is tokenized by Senten
Transformer Decoder	w/o PE	w/ PE
WikiText-103	23.52	23.37
Chinese Wiki	12.96	12.75
Table 1: Transformer Decoders perplexity (PPL) on WikiText-103 and Chinese Wikipedia validation sets.  ${PE}$  refers to the learnable position embeddings.
cePiece tokenizer (Kudo and Richardson, 2018), which is the same as RoBERTa. All Chinese data is tokenized by character.
All models are trained with Fairseq. The training objective is the Causal Language Modeling objective. We use a batch size of 128 and train for 100k steps, optimized by Adam (Kingma and Ba, 2015). We also use the polynomial learning rate decay with 10k warmup steps. All models use the same hyper-parameters. We list the details in the Appendix. We use two NVIDIA A100 40GB GPUs to train each model. For the WikiText-103, it costs about 10 hours per model. For the Chinese Wikipedia, it costs about 8.5 hours per model.
Results and Discussion. Table 1 presents the perplexity (PPL) scores of Transformer Decoders with or without position embeddings on WikiText-103 and Chinese Wikipedia validation sets. Transformer Decoder w/o PE achieves comparable performance with its counterpart with learnable PE, which is only about 0.2 higher. This result reveals that the additional performance gain brought by position embeddings is small. Only relying on its causal attention masks, Transformer Decoder still can perform well. Combining our experiment and the previous studies (Tsai et al., 2019; Irie et al., 2019),
2The Chinese vocabulary size is smaller than English, so the Chinese model has fewer parameters.
the causal attention masks can make Transformer sensitive to word order.
3.2 Our DecBERT Model
In section 3.1, we conclude that Transformer with the causal attention masks is naturally sensitive to word order. Since the position information is inevitable for BERT, we propose to enhance existing BERT model based on causal attention masks.
In this paper, we intend to add the causal attention masks into all or some hidden layers of BERT. In this way, the specific layers with such masks are sensitive to word order by design, which can enhance the position encodings ability of BERT. Such framework can further result in better language understanding performances, e.g., in pretrained language modeling, casual attention masks were added on all 12 layers of GPT (Radford and Narasimhan, 2018). However, comparing with BERT (Devlin et al., 2019), we observe that GPT lags behind BERT on almost all downstream tasks. This is because self-attention mechanism with such masks only consider one-side information flow, it cannot process the input sentence comprehensively and has a high risk of language information loss. Therefore, we can conjecture that it is not effective to use the causal attention masks in all hidden layers. There is a strong need to maintain a balance between the gain of position encoding ability and the loss of language information.
In order to determine which layer(s) should add casual attention masks, we refer to the BERTology work (Jawahar et al., 2019) that conduct comprehensive experiments to analyze and interpret the information captured by each layer of BERT. The experimental results indicate that the lower layers of BERT capture rich language structure information. The position information is also a common structure information, so that we propose to add the causal attention masks into the lower layers (e.g., the first two layers<sup>4</sup>) to improve the position encoding ability of BERT. We denote our model as DecBERT. There are two versions of our model, DecBERT-Same and DecBERT-Diff. All of them are 12-layer base size models.
- DecBERT-Same: This model has a similar
structure as BERT (see Figure 1(a)), but we use the causal attention masks to convert the first two Encoder layers into two Decoder layers with the same direction (from left to right). So the 12-layer model has 10 Encoder layers and 2 Decoder layers, which is shown in Figure 1(b). In this way, the first two layers are naturally sensitive to word order;
- DecBERT-Diff: This model is designed to enhance DecBERT-Same to gain more language information from different encoding directions. This model has a same structure as DecBERT-Same, except the second Decoder layer that has the opposite direction (from right to left). Figure 1(c) illustrates the model structure.
One would think that DecBERT is similar to Transformer with RNN layer (Neishi and Yoshinaga, 2019). Note that DecBERT is quite different from it, because DecBERT has similar structure as BERT and both of them require the same amount of computational time, which is much faster than that of Transformer with RNN.
Our experiments can be separated into two parts, small-scale pre-training scenario and large-scale pre-training scenario. Since the small-scale pretraining consumes much less time and fewer computational resources, we intend to answer several research questions in this part:
- Can DecBERT without any position embeddings still understand language well?
- Can DecBERT with position embeddings outperform BERT?
- Is using different directional causal attention masks more helpful than using the same directional?
- Why can DecBERT benefit from the causal attention masks, how do such masks affect the pre-training process?
For the large-scale pre-training scenario, we intend to examine whether the performance gap between our DecBERT and BERT will be diminished after scaling up the pre-training data size and time. Such settings can present a more comprehensive
3 Although GPT and BERT are pre-trained with different objectives, the comparison is reasonable due to the same downstream tasks.
4We conducted massive experiments by adding the masks in the first, first-two, or first-three layer(s), and the first-two layers achieve the best performance.
view of whether our modification can benefit the pre-trained language models.
For a fair comparison, we re-implement BERT and pre-train it with the same settings as DecBERT in the small-scale and large-scale pre-training. We denote it as BERT-reImp.
Small-scale Pre-training Scenario. The pretraining data is the widely-used English Wikipedia Corpus. We randomly select 158.4M sentences for training and 50k sentences for validation. The pre-training objective is the Masked Language Modeling objective. We use a batch size of 256 and pre-train for 200k steps, optimized by Adam. All models use the same hyper-parameters. We list the details in the Appendix. We use four NVIDIA A100 40GB GPUs to pre-train each model, costing about 34.5 hours per model.
Large-scale Pre-training Scenario. Limited by time and computational resources, it is impossible for us to pre-train all models in the small-scale pre-training scenario from scratch in this setting. Thus, we decide to pre-train the best model in the small-scale scenario and the baseline model BERT-reImp w/ PE in this part. We use a large amount of pre-training data (around  $160\mathrm{GiB}^5$ ). The batch size is set to 4096 and the pre-training steps are 300k. We pre-train each model with 8 NVIDIA A100 40GB GPUs, costing about 15 days per model. The hyper-parameters details can be also seen in the Appendix.
Fine-tuning. To evaluate the language understanding ability of our models, we fine-tune them with 8 tasks of GLUE benchmark (Wang et al., 2019b), including SST-2 (Socher et al., 2013), QNLI (Rajpurkar et al., 2016), MNLI (Williams et al., 2018), QQP, $^{6}$  MRPC (Dolan and Brockett, 2005), CoLA (Warstadt et al., 2019), RTE $^{7}$  and STS-B (Cer et al., 2017). All fine-tuning hyperparameters details are listed in the Appendix.
4.2 Small-scale Pre-training
Table 2 presents the pre-training perplexity scores of all systems on the validation set. Table 3 shows the performance of different systems on the GLUE
Models	w/ PE	Valid PPL
Baseline
BERT-reImp	False	353.97
BERT-reImp	True	4.28
Ours (w/o position embeddings)
DecBERT-Same	False	4.59
DecBERT-Diff	False	4.59
Ours (w/ position embeddings)
DecBERT-Same	True	4.12
DecBERT-Diff	True	4.07
Table 2: The validation set perplexity of all models in small-scale pre-training scenario. (w/  $PE =$  with learnable position embeddings)
benchmark. One can notice that our proposed models achieve lower valid PPL scores and higher overall scores on the downstream tasks.
Can DecBERT without any position embeddings still understand language well? Since the self-attention of Transformer Encoder is order-invariant, the extra position information is inevitable for it to model language. Otherwise, it just becomes a bag-of-word model. From Table 2, we can find that the valid PPL score of BERT-reImp w/o PE is up to 353.97, which is about 82 times higher than its counterpart with position embeddings (4.28), revealing that this bag-of-word model cannot model language well. However, one can notice that DecBERT does not have such phenomenon. The valid PPL score of DecBERT w/o PE is only about 0.5 higher than DecBERT w/ PE. Compared with BERT-reImp w/o PE, the causal attention masks can decrease the PPL score by a large margin (from 353.97 to 4.59). After fine-tuning on downstream tasks, Table 2 indicates that DecBERT-Same/Diff w/o PE retains the same level performance as BERT-reImp w/ PE. These results reveal that DecBERT still can understand language well without the help of position embeddings, which is in line with our experimental results in section 3.1.
Can DecBERT with position embeddings outperform BERT? Table 2 shows that both DecBERT-Same and DecBERT-Diff have lower validation PPL scores than BERT-reImp (w/ PE). After fine-tuning on the downstream tasks, Table 3 reveals that they also have better performance on most tasks. These results confirm our belief that
5The details of our pre-training corpus can be seen in the Appendix. 6https://www.quora.com/q/quoradata/First-QuoraDataset-Release-Question-Pairs 7https://aclweb.org/aclwiki/Recognizing_Textual_Entailme
Models	SST-2	QNLI	QQP	RTE	MNLI-m/mm	MRPC	STS-B	Avg.
Small-scale pretraining results on the dev sets
BERT-reImp	89.56	89.24	90.14	64.40	80.14/80.62	86.60	86.22	83.37
Ours (w/o position embeddings)
DecBERT-Same	89.58	89.50	90.16	62.68	79.56/80.42	85.88	86.58	83.05
DecBERT-Diff	90.30	88.86	90.28	59.28	79.78/80.78	86.08	86.06	82.68
Ours (w/ position embeddings)
DecBERT-Same	90.12	89.18	90.32	64.78	80.48/80.64	86.24	86.34	83.51
DecBERT-Diff	90.78	89.56	90.08	65.98	80.92/81.26	85.86	86.24	83.84
Table 3: Different small-scale pre-training models' performance on the dev sets of GLUE benchmark. All results are averaged over five different random seeds (1, 2, 3, 4 and 5). MNLI-m is the matched version and MNLI-mm is the mismatched version. All tasks except STS-B use accuracy as their evaluation metrics. STS-B uses the Spearman rank correlation. The results are reported as  $r \times  {100}$  . Bold indicates the best score for each task.
Models	SST-2	QNLI	QQP	RTE	MNLI-m/mm	CoLA	MRPC	STS-B	Avg.
Large-scale pretraining results on the test sets
BERT-reImp	94.7	91.5	89.4	66.5	85.9/85.1	56.3	85.4	86.8	82.4
DecBERT-Diff	94.5	92.0	89.3	72.0	86.8/85.5	59.6	86.0	86.8	83.6
Table 4: Different large-scale pre-training models' performance on the test sets of GLUE benchmark. All tasks except STS-B and CoLA use accuracy as their evaluation metrics. STS-B uses the Spearman rank correlation. CoLA uses the Matthews correlation coefficient. The results are reported as  $r \times 100$ . Bold indicates the best score of our models for each task.
our models can benefit from the causal attention masks. Such masks enhance the position encoding ability of BERT, leading to better language understanding ability.
Is using different directional causal attention masks helpful? The only difference between DecBERT-Same and DecBERT-Diff is that we adopt a different directional causal attention mask in the second layer. Table 2 shows that DecBERT-Diff w/ PE achieves the lowest validation PPL score (4.07). After fine-tuning on the downstream tasks, it also has the best overall score. These results confirm our belief that DecBERT can benefit from using different directional attention masks. Though the first two layers of DecBERT-Diff only consider one-side information flow, the model can learn to process different directional information in the first two layers. This design maintains a better balance between the gain of position encoding ability and the loss of language information.
Why can DecBERT benefit from the causal attention masks? The experimental results in the previous part indicate that the causal attention masks can increase the model's position encoding ability. Then such ability leads to better lan
Figure 2: The pre-training loss of the first 16k steps. (Small-scale pre-training)
guage understanding ability. However, the relation between these two abilities remains unclear. We analyze the pre-training process of our models to give a possible explanation.
Our models' pre-training loss curves are presented in Figure 2 and 3. Since the randomly initialized Multi-head Self-Attention of BERT is a "balance" structure without any inductive bias, the model needs to learn suitable position embeddings to trace the word order during pre-training. In Figure 2, one can notice that the pre-training process of BERT-reImp w/ PE can be divided into four stages: (1) starting stage (0-1000 steps), (2) plateau
Figure 3: The pre-training loss of the last 120k steps. (Small-scale pre-training)
stage (1000-8000 steps), (3) "diving" stage (8000-10000 steps) and (4) convergence stage (10000-final steps). In the starting and plateau stages, BERT-reImp w/ PE has almost the same training loss as its counterpart without PE, which indicates that it is still a bag-of-word model and does not know how to make use of the position information. In the "diving" stage, the training loss of BERT-reImp w/ PE decreases rapidly, while BERT-reImp w/o PE starts to converge. This reveals that the word order information becomes more useful for models to understand language in such stage. In the convergence stage, the training loss decreases slowly to the end of the whole pre-training process.