Before revision:  We are the first to identify structured winning tickets in the early stage of BERT training , and use them for efficient training.
After revision:  By slimming the self-attention and fully-connected sub-layers inside a transformer, we are the first to identify structured winning tickets in the early stage of BERT training , and use them for efficient training.