Before revision:  For model pre-training, we propose the masked sentence block language modeling task in addition to the original masked word language modeling task used in BERT, to capture sentence block relations within a document.
After revision:  For model pre-training, we propose the masked sentence block language modeling task in addition to the masked word language modeling task used in BERT, to capture sentence block relations within a document.