Before revision:  However, PLMs still cannot well capture the factual knowledge in the text, which is crucial for understanding the whole text, especially for document-level language understanding tasks .
After revision:  However, conventional pre-training objectives do not explicitly model relational facts in text, which is crucial for understanding the whole text, especially for document-level language understanding tasks .