Before revision:  To verify this, we propose a segment-aware BERT , by replacing the token position embedding of Transformer with a combination of paragraphindex, sentenceindex ,  and token index embeddings. We pre-trained the SegaBERT on the masked language modeling task in BERT but without any affiliated tasks . Experimental results show that our pre-trained model can outperform the original BERT model on various NLP tasks  .
After revision:  To verify this, we propose a segment-aware BERT , by replacing the token position embedding of Transformer with a combination of paragraphindex, sentenceindex , and token . We first introduce the segment-aware mechanism to Transformer-XL, which is a popular Transformer-based language model with memory extension and relative position encoding. We find that our method can further improve the Transformer-XL base model and large model, achieving 17.1 perplexity on the WikiText-103 dataset. We further investigate the pre-training masked language modeling task in BERT but without any affiliated tasks . Experimental results show that our pre-trained model can outperform the original BERT model on various NLP tasks .