Before revision:  In light of our analyses, the proposed method CompoundGrow speeds up BERT pre-training by 73.6\% and 82.2\% for the base and large models respectively  while achieving comparable performances . Code will be released for reproduction and future studies.
After revision:  In light of our analyses, the proposed method CompoundGrow speeds up BERT pre-training by 73.6\% and 82.2\% for the base and large models respectively while achieving comparable performances