Before revision:  We are the first to identify structured winning tickets in the early stage of BERT training , and use them for efficient training. Comprehensive pre-training and fine-tuning experiments on GLUE and SQuAD downstream tasks  show that EarlyBERT easily achieves comparable performance to standard BERT  with 35~45\% less training time. 
After revision:  We are the first to identify structured winning tickets in the early stage of BERT training . We apply those tickets towards efficient BERT training, and conduct comprehensive pre-training and fine-tuning experiments on GLUE and SQuAD downstream tasks show that EarlyBERT easily achieves comparable performance to standard BERT with 35~45\% less training time.