Before revision:  We pre-trained the SegaBERT on the masked language modeling task in BERT but without any affiliated tasks . Experimental results show that our pre-trained model can outperform the original BERT model on various NLP tasks  .
After revision:  We pre-trained the SegaBERT on the masked language modeling task in BERT but without any affiliated tasks . Experimental results show that our pre-trained with Segatron (SegaBERT) can outperform BERT with vanilla Transformer on various NLP tasks .