Before revision:  We pre-trained the SegaBERT on the masked language modeling task in BERT but without any affiliated tasks . Experimental results show that our pre-trained model can outperform the original BERT model on various NLP tasks  .
After revision:  We pre-trained the SegaBERT on the masked language modeling task in BERT but without any affiliated tasks . Experimental results show that our pre-trained model can outperform the original BERT model on various NLP tasks , and outperforms RoBERTa on zero-shot sentence representation learning .