SlowGuess/NAACL_2022_OCR huggingface dataset:
So, how do the causal attention masks affect the pre-training process? The first two layers of DecBERT can break the "balance" of the multihead self-attention by design. The position bias from the attention masks makes the first two layers sensitive to word order information. In Figure 2, one can notice that the plateau stage of DecBERT is shortened (from around 7000 to 3000 steps). This reveals that DecBERT does not need to spend as much time as BERT to learn to make use of the position information. It can escape from the bag-of-words sub-optimal point faster. Though the gap between BERT-reImp w/ PE and DecBERT-Diff w/ PE become smaller in the convergence stage, Figure 3 indicates that DecBERT-Diff w/ PE still has lower training loss in the whole pre-training process.
4.3 Large-scale Pre-training
In the large-scale pre-training scenario, we intend to verify whether our modification still achieves better performance. From Figure 4 and Table 4, one can find that the experimental results are similar to
Figure 4: The PPL scores on validation set from epoch 5 to epoch 15 of our models. (Large-scale pre-training)
the small-scale pre-training scenario. For the validation PPL, DecBERT-Diff achieves lower scores than BERT-reImp in the whole pre-training process. Especially, at the  $13^{th}$  epoch (265k steps), the valid PPL score of DecBERT-Diff is 3.48, which is the same as BERT-reImp at the  $15^{th}$  epoch (300k steps). This suggests that the pre-training process of DecBERT-Diff is about 2 epochs faster than BERT-reImp. Combining our previous analysis, one advantage of our modification is that it can accelerate the pre-training process. Comparing the downstream tasks, one can also notice that the performance gap between DecBERT-Diff and BERT-reImp even becomes larger. The average score is 1.2 points higher.
All results in this part indicate that our modification is effective not only in the small-scale pretraining, but also in the large-scale pre-training. It can accelerate the pre-training process. When pretraining with the same amount of computational resources, our modification can achieve better performance on masked language modeling and downstream tasks.
The analysis and experimental results detailed in the previous sections point out an interesting finding that the pre-training process of BERT can be divided into different stages. A similar phenomenon also can be found in the work of Kovaleva et al. (2021). In their work, they find that both scaling factors and biases of the Layer Normalization begin to diverge from their initialization values quickly in the "diving" stage. Especially, one/two specific neurons of the biases have larger and larger absolute values. Luo et al. (2021) indicates that such neurons are highly related to the positional informa
Figure 5: The pre-training loss of the first 40k steps. (Extra small-scale pre-training)
tion. These complement our possible explanation that in the plateau stage, the model needs to learn suitable position embeddings. Then in the "diving" stage, the model learns to adopt such embeddings to better model language. Our DecBERT models indicate that breaking the "balance" by design can help BERT better capture the position information, which leads to better performance.
One would wonder how about the fixed sinusoidal position embeddings. With such embeddings, BERT does not need to learn suitable position embeddings during pre-training. Based on our previous analysis, the plateau stage is possible to disappear. To examine whether such position embeddings are better, we conduct an extra small-scale pre-training experiment. The pre-training loss curve is in Figure 5, revealing that the plateau stage indeed disappears. This is in line with our previous results. However, in the convergence stage, we find that BERT with the sinusoidal PE has higher pre-training loss than using the learnable PE. This indicates that the learnable position embeddings are more suitable for BERT.
The previous works (Vaswani et al., 2017; Shaw et al., 2018; Huang et al., 2019; Dai et al., 2019; Child et al., 2019) indicate that the self-attention mechanism of Transformer Encoder is permutation equivalent, so it needs to use the position embedding. Tsai et al. (2019) have proved that Decoder's self-attention is not permutation equivalent, indicating that Decoder is not a bag-of-word model as Encoder, but they do not conduct further analysis on Decoder's position encoding ability. Apart from the analysis, Irie et al. (2019) train the Transformer Language Models with speech dataset. They find
that models without position embeddings have lower perplexity scores. Schlag et al. (2021a) introduce a new Linear Transformer Language Model with fast weight memories (Schmidhuber, 1992; Schlag et al., 2021b), which has lower perplexity without position encodings on the WikiText-103 dataset.
Furthermore, an explosion of work focuses on proposing a better method to add the position information into the pre-trained language model. Dufter et al. (2021) give a comprehensive introduction of different position encodings methods of Transformer. They divide position encodings into three approaches. One line of such work is to add position embeddings to the input before it is fed to the actual Transformer model (Vaswani et al., 2017; Shaw et al., 2018; Devlin et al., 2019; Kitaev et al., 2020; Liu et al., 2020; Press et al., 2020; Wang et al., 2020). The second line of work directly modify the attention matrix (Dai et al., 2019; Dufter et al., 2020; He et al., 2020; Wu et al., 2021a; Ke et al., 2021; Su et al., 2021). The last one combines the first two approaches together. However, all of them focus on introducing an extra set of parameters to trace the word order. Our work chooses to make use of the causal attention masks.
Most similar to our modification in Section 3.2, Im and Cho (2017) propose a self-attention based model which achieve better performance on SNLI task (Bowman et al., 2015) without the help of explicit position encodings. However, their models are different from the standard Transformer and use extra local attention masks to control the information flow. With the popularity of the Transformer model in the Computer Vision field, some works propose different methods to make Vision Transformer know word order implicitly (Chu et al., 2021; Yuan et al., 2021; Wu et al., 2021b), but all of them modify the models with convolution neural network (Lecun et al., 1998).
In this work, we introduce a new pre-trained model, called DecBERT, adopting the causal attention masks to enhance the language understanding of BERT. We conduct a series of experiments to verify the effectiveness of our models. Experimental results indicate that our proposed models achieve better performance than BERT on most downstream tasks when pre-training with the same amount of data and computational resources. Moreover, our
analysis also indicates that our models can accelerate the pre-training process.
We would like to thank Artur Kulmizev and the anonymous reviewers for their excellent feedback. This work is supported by the Key Research and Development Program of Zhejiang Province (No. 2022C01011), HKBU One-off Tier 2 Startup Grant (Ref. RCOFSGT2/20-21/SCI/004) and HKBU direct grant (Ref. AIS 21-22/02).
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1-14, Vancouver, Canada. Association for Computational Linguistics.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers.
Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. 2021. Conditional positional encodings for vision transformers.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond
a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988, Florence, Italy. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Philipp Dufter, Martin Schmitt, and Hinrich Schütze. 2020. Increasing learning efficiency of self-attention networks through direct position interactions, learnable temperature, and convoluted attention. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3630-3636, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Philipp Dufter, Martin Schmitt, and Hinrich Schütze. 2021. Position information in transformers: An overview.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The pile: An 800gb dataset of diverse text for language modeling.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention.
Dan Hendrycks and Kevin Gimpel. 2020. Gaussian error linear units (gelus).
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735-1780.
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. 2019. Music transformer. In International Conference on Learning Representations.
Jinbae Im and Sungzoon Cho. 2017. Distance-based self-attention network for natural language inference.
Kazuki Irie, Albert Zeyer, Ralf Schlüter, and Hermann Ney. 2019. Language modeling with deep transformers. Interspeech 2019.
Ganesh Jawahar, Benoit Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651-3657, Florence, Italy. Association for Computational Linguistics.
Guolin Ke, Di He, and Tie-Yan Liu. 2021. Rethinking positional encoding in language pre-training. In International Conference on Learning Representations.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. 2021. Bert busters: Outlier dimensions that disrupt transformers.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. 2019. Set transformer: A framework for attention-based permutation-invariant neural networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3744-3753. PMLR.
Xuanqing Liu, Hsiang-Fu Yu, Inderjit S. Dhillon, and Cho-Jui Hsieh. 2020. Learning to encode position for transformer with continuous dynamical model. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 6327-6335. PMLR.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
Ziyang Luo, Artur Kulmizev, and Xiaoxi Mao. 2021. Positional artefacts propagate through masked language model embeddings. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5312-5327, Online. Association for Computational Linguistics.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open-Review.net.
Masato Neishi and Naoki Yoshinaga. 2019. On the relation between position information and sentence length in neural machine translation. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 328-338, Hong Kong, China. Association for Computational Linguistics.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairoseq: A fast, extensible toolkit for sequence modeling*. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics* (Demonstrations), pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Ofir Press, Noah A. Smith, and Mike Lewis. 2020. Shortformer: Better language modeling using shorter inputs.
A. Radford and Karthik Narasimhan. 2018. Improving language understanding by generative pre-training.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. 2021a. Linear transformers are secretly fast weight memory systems.
Imanol Schlag, Tsendsuren Munkhdalai, and Jürgen Schmidhuber. 2021b. Learning associative inference using fast weight memory. In International Conference on Learning Representations.
Jürgen Schmidhuber. 1992. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131-139.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464-468, New Orleans, Louisiana. Association for Computational Linguistics.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. 2021. Roformer: Enhanced transformer with rotary position embedding.
Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Transformer dissection: An unified understanding for transformer's attention via the lens of kernel. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4344-4353, Hong Kong, China. Association for Computational Linguistics.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 3261-3275.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Benyou Wang, Donghao Zhao, Christina Lioma, Qiuchi Li, Peng Zhang, and Jakob Grue Simonsen. 2020. Encoding word order in complex embeddings. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
Chuhan Wu, Fangzhao Wu, and Yongfeng Huang. 2021a. Da-transformer: Distance-aware transformer.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. 2021b. Cvt: Introducing convolutions to vision transformers.
Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. 2021. Incorporating convolution designs into visual transformers.
Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.
A Hyper-parameters Details
Hyper-parameter	w/ or w/o PE
Number of Layers	8
Hidden size	768
FNN inner hidden size	3072
Attention Heads	12
Attention Head size	64
Dropout	0.1
Warmup Steps	10k
Max Steps	100k
Learning Rates	5e-5
Batch Size	128
Weight Decay	0.001
Learning Rate Decay	Polynomial
Adam ε	1e-6
Adam β1	0.9
Adam β2	0.998
Gradient Clipping	0.1
Random Seed	1
Table 5: Hyper-parameters for pre-training the Transformer Decoder Causal Language Models.
Hyper-parameter	BERT/DecBERT
Number of Layers	12
Hidden size	768
FNN inner hidden size	3072
Attention Heads	12
Attention Head size	64
Dropout	0.1
Warmup Steps	24k
Max Steps	500k
Learning Rates	3e-4
Batch Size	4096
Weight Decay	0.01
Learning Rate Decay	Tri stage
Adam ε	1e-6
Adam β1	0.9
Adam β2	0.98
Gradient Clipping	2.0
Table 7: Hyper-parameters for pre-training the BERT and DecBERT (large-scale pre-training).
Hyper-parameter	BERT/DecBERT
Number of Layers	12
Hidden size	768
FNN inner hidden size	3072
Attention Heads	12
Attention Head size	64
Dropout	0.1
Warmup Steps	10k
Max Steps	200k
Learning Rates	1e-4
Batch Size	256
Weight Decay	0.01
Learning Rate Decay	Polynomial
Adam ε	1e-6
Adam β1	0.9
Adam β2	0.98
Gradient Clipping	0.5
Random Seed	1
Table 6: Hyper-parameters for pre-training the BERT and DecBERT (small-scale pre-training).
B The details of the large-scale pre-training corpus
The first part is the same as BERT. We use the English wikipedia dump (about 17 GiB) and the bookcorpus (Zhu et al., 2015) (about 4 GiB). The second part is based on the Pile dataset (Gao et al., 2020), which is a large datasets with 800 GiB diverse text data. We randomly extract 64 GiB data
from the Pile-cc block, 35 GiB data from the OpenWebText2 block and 43 GiB data from the Books3 block. The overall size of all data is about 163 GiB.
Hyper-parameter	MNLI	QNLI	QQP	RTE	SST-2	MRPC	STS-B	CoLA
Learning Rates	1e-5	1e-5	1e-5	2e-5	1e-5	{1e-5, 2e-5}	2e-5	1e-5
Weight Decay	0.01	0.01	0.01	0.01	0.01	0.01	0.01	0.01
Batch Size	32	32	32	16	32	16	16	16
Warmup Steps	7432	1986	28318	122	1256	137	214	320
Max Steps	123873	33112	113272	2036	20935	2296	3598	5336
Adam ε	1e-6	1e-6	1e-6	1e-6	1e-6	1e-6	1e-6	1e-6
Adam β1	0.9	0.9	0.9	0.9	0.9	0.9	0.9	0.9
Adam β2	0.999	0.999	0.999	0.999	0.999	0.999	0.999	0.999
Gradient Clipping	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
Table 8: Hyper-parameters for fine-tuning all models on downstream tasks. All models use the polynomial learning rate decay. Most of the hyper-parameters are recommended by Fairseq https://github.com/pytorch/fairseq/tree/main/examples/roberta/config/binfetuning.