SlowGuess/NAACL_2021_OCR huggingface dataset:
Liyuan Liu, Haoming Jiang, Pengcheng He, W. Chen, Xiaodong Liu, Jianfeng Gao, and J. Han. 2020a. On the variance of the adaptive learning rate and beyond. In ICLR.
Liyuan Liu, X. Liu, Jianfeng Gao, Weizhu Chen, and J. Han. 2020b. Understanding the difficulty of training transformers. In EMNLP.
Liyuan Liu, Jingbo Shang, Xiang Ren, Frank Fangzheng Xu, Huan Gui, Jian Peng, and Jiawei Han. 2018. Empower sequence labeling with task-aware neural language model. In AAAI.
Y. Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In *NAACL-HLT*.
A. Radford. 2018. Improving language understanding by generative pre-training.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. In ACL.
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. In *ICLR*.
Mingxing Tan and Quoc Le. 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, L. Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In ICLR.
Tao Wei, Changhu Wang, and Chang Wen Chen. 2017. Modularized morphing of neural networks. arXiv preprint arXiv:1701.03281.
Tao Wei, Changhu Wang, Yong Rui, and Chang Wen Chen. 2016. Network morphism. In ICML.
Wei Wen, Feng Yan, Yiran Chen, and Hai Li. 2020. Autogrow: Automatic layer growing in deep convolutional networks. In KDD.
Chao-Yuan Wu, Ross Girshick, Kaiming He, Christoph Feichtenhofer, and Philipp Krahenbuhl. 2020. A multigrid method for efficiently training video models. In CVPR.
Figure 2: Compare the speed-performance trade-off of stacking and CompoundGrow on  $\mathrm{BERT}_{\mathrm{large}}$ . The three data points in each curve is generated with  $300\mathrm{K} / 500\mathrm{K} / 700\mathrm{K}$  low-cost training steps, respectively.
All our models are implemented based on the TensorFlow implementation<sup>3</sup> of BERT (Chen et al., 2020) and trained on TPU v3 with 64 chips. We keep the original WordPieceTokenizer and original position embeddings (instead of relative position encoding used in Dai et al., 2020). Following Devlin et al., 2019), we use the English Wikipedia corpus and the BookCorpus for pre-training. For each finetuning task, we search hyperparameters from following candidates: batch size  $= 16 / 32 / 64$ , learning rate  $= 3e - 4 / 1e - 4 / 5e - 5 / 3e - 5$
Optimization. The original BERT models use the AdamW (Loshchilov and Hutter, 2019) optimizer with learning rate decay from 0.0001 to 0 and 10K steps of warmup (Liu et al., 2020a). At the start of each progressive training stage, the learning rate is reset to 0.0001 and keeps decaying as the original schedule.
Baseline Implementation. We apply the compared stacking method (Gong et al., 2019) on the official BERT model with the same training setting, learning rate schedule and hardware as our method, and achieves better performance than the reported numbers in the original paper. To further unleash the potential of the compared method, we adjust their original training schedule to  $300\mathrm{K}$  steps with  $\frac{1}{4}$  number of layers,  $400\mathrm{K}$  steps with  $\frac{1}{2}$  number of layers, and  $300\mathrm{K}$  steps with the full model. The new training schedule is much faster than the reported one (speedup from the reported  $+25\%$  to  $+64.9\%$ ) and still gives better final performance than the original paper. This is the fastest stacking model we can get without performance drop.
B Further Comparison Between CompoundGrow and Stacking
To have a deeper understanding of the compared methods, we study their speed-performance tradeoff by adjusting the training schedule. Specifically, each time we reduce 200K low-cost training steps for both models, and compare their validation F1 score on SQuaDv2.0. As Figure 2 shows, CompoundGrow has clear performance advantage when given comparable training budgets, which further verifies our hypothesis.
$^{3}$ https://github.com/tensorflow/models/blob/master/official/nlp/modeling/models/bert_pretrainer.py