SlowGuess/NAACL_2021_OCR huggingface dataset:
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. 2018. Sharp nearby, fuzzy far away: How neural language models use context. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 284-294, Melbourne, Australia. Association for Computational Linguistics.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Swetha Mandava, Szymon Migacz, and Alex Fit Florea. 2020. Pay attention when required. arXiv preprint arXiv:2009.04534.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2017. Regularizing and optimizing LSTM language models. CoRR, abs/1708.02182.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. An analysis of neural language modeling at multiple scales. CoRR, abs/1803.08240.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. CoRR, abs/1609.07843.
Tomáš Mikolov, Stefan Kombrink, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur. 2011. Extensions of recurrent neural network language model. In 2011 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5528-5531. IEEE.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1525-1534, Berlin, Germany. Association for Computational Linguistics.
Ofir Press, Noah A. Smith, and Omer Levy. 2020a. Improving transformer models by reordering their sublayers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2996-3005, Online. Association for Computational Linguistics.
Ofir Press, Noah A. Smith, and Mike Lewis. 2020b. Shortformer: Better language modeling using shorter inputs.
Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 157-163, Valencia, Spain. Association for Computational Linguistics.
Aurko Roy, M. Saffar, Ashish Vaswani, and David Grangier. 2020. Efficient content-based sparse attention with routing transformers. ArXiv, abs/2003.05997.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929-1958.
Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. 2019. Adaptive attention span in transformers. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 331-335, Florence, Italy. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998-6008. Curran Associates, Inc.
Chenguang Wang, Mu Li, and Alexander J. Smola. 2019. Language models with transformers. CoRR, abs/1904.09408.
John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Towards universal paraphrastic sentence embeddings. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ot-atanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33.
Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization.
A Experiment details
Dataset	Train #Tokens	Vocab. size
WIKITEXT-2	2M	33K
WIKITEXT-103	103M	267K
LAMBADA	203M	60K
ENWIK8	100M	205
Table 4: Dataset statistics
Dataset statistics are shown in Table 4.
Dataset	Train len	Test len	Tgt. len
WIKITEXT-2	512	512	128
WIKITEXT-103	512	512	128
LAMBADA	512	512	128
ENWIK8	1024	1024	512
Table 5: Training sequence length as well as scored target length and total test sequence length during evaluation we used on each dataset.
Evaluation We follow the practice in (Khandelwal et al., 2020) to provide extra prior context for the scored tokens. We provide the training sequence length, test total sequence length, and test target sequence length in Table 5.
B Model configurations
Detailed model configurations are shown in Table 6. Training details are shown in Table 7.
WIKITEXT-2	WIKITEXT-103	ENWIK8	LAMBADA
NPLM	Transformer	NPLM	Transformer	NPLM	Transformer	NPLM	Transformer
# Layers	6	6	16	16	12	12	16	16
Emb. dimension	256	256	410	410	512	512	512	512
Hidden dimension	1024	1024	2100	2100	2048	2048	4096	4096
Concat hidden dimension	400	-	2000	-	1400	-	2000	-
# Attention heads	-	4	-	10	-	8	-	16
Adaptive softmax	no	no	yes	yes	no	no	no	no
# Concat tokens	15	-	15	-	15	-	15	-
# Kernel global	5	-	5	-	5	-	5	-
Dropout	0.3	0.3	0.2	0.1	0.2	0.1	0.2	0.1
#Param	13M	13M	149M	148M	38M	38M	115M	115M
Table 6: Model configuration on WIKTEXT-2, WIKTEXT-103, ENWIK8, LAMBADA.
Warmup steps	Learning rate	Max steps	Batch size	Training time
WIKTEXT-2	100	5e-4	10k	5120	1.2h/1h
WIKTEXT-103	4k	2.5e-4/3.5e-4	200k	10240	40h/32h
ENWIK8	0	2.5e-4	400k	22528	102h/45h
LAMBADA	4k	3e-4	400k	8192	108h/88h
Table 7: Details of training on the four datasets. Models are trained on single 1080Ti GPU for WIKITEXT-2, and on four 1080Ti GPUs for the rest datasets. When a configuration is different for Transformer and NPLM, it's shown in the order Transformer/NPLM.