Before revision:  In this paper, we present a new approach to improve the capability of language model pre-training on VRDs. Firstly, we introduce a new  IE model that is query-based and employs the span extraction formulation instead of the commonly used sequence labelling approach.
After revision:  In this paper, we present a new approach to improve the capability of language model pre-training on VRDs. Firstly, we introduce a new IE model that employs span extraction instead of using the common sequence labeling approach.