## 2 Related Work  
Pre-trained Language Model. Early research on pre-training neural representations of language mainly attends to pre-trained context-free word embeddings, such as Word2 Vec [11], GloVe [12]. Since most Natural Language Processing (NLP) tasks are beyond word level, researchers propose to pre-train sequential encoders, such as LSTM, to obtain contextual word embeddings [13]. Recently, the very deep pre-trained language models (PLMs) [5; 6] have improved the performance on multiple natural language processing benchmarks [14], which is credited to the strong prior knowledge and the deep network structure. OpenAI GPT [15] proposes the first transformer-based PLM with a left-to-right language modeling and auto-encoder objective. BERT [5] improves GPT by using a Masked Language Model (MLM) pre-training objective to learn bidirectional encoder representations, and a Next Sentence Prediction (NSP) objective to learn the relation between sentence pairs. BERT encodes diverse information in different layers. [PERSON] et al. [8] conduct a series of experiments to unpack the elements of English language structure learned by BERT. This work finds that BERT captures structural information about language. It learns phrase-level information and surface information in the low layer, syntactic features in the middle layer and semantic features at the top layer. Moreover, [PERSON] et al. [16] also find that the middle layer of BERT, rather than the top layer, performs best on syntax task. Following GPT and BERT, a lot of transformer-based PLMs are proposed. These models use more data [6] and parameters [17], modified word masking methods [6; 18] and transformer structures [19; 20] to significantly improve the SOTA performance on multiple NLP datasets. Machine Reading Comprehension. Early Machine Reading Comprehension (MRC) systems use rules-based methods or feature engineering methods to extract answers from passages. Deep Read [21] proposes the first reading comprehension system. Given a story and a question, it uses sentence matching (bag-of-words) techniques and some linguistic processing methods, such as stemming and pronoun resolution, to select a sentence from the story as the answer to the question. Quarc [22] and Cqarc [23] are two rules-based reading comprehension systems which design some hand-crafted heuristic rules to look for the semantic clues and answer the given questions. These methods depend on human-defined feathers and are difficult to generalize to new datasets [24]. The rapid growth of MRC is largely thanks to the development of deep learning [25; 26] and the availability of large-scale datasets [1; 2]. The attentive reader [26] proposes the first neural MRC models based on LSTM and attention mechanism, in which two cloze-style corpuses are also presented. Then the attentive reader is extended and enhanced in several ways. For the embedding module, BIDAF [27] introduces char-level embedding into the model that alleviates the out-of-vocabulary (OOV) problem. DrQA [28] improves the embedding representations of both question and passage with POS embedding, NER embedding and binary feature of exact match. For the encoder, QANet [29] replaces the LSTM encoder with convolution network to accelerate the training and inference processes. To better fuse the question and passage, BIDAF [27] proposes a bi-directional attention flow mechanism and GA [30] presents a static multi-hop attention architecture, both of which obtain a fine-grained question-aware passage representation. DFN [31] proposes a sample-specific network architecture with dynamic multi-strategy attention process, lending flexibility to adapt to various question types that require different comprehension skills. Hierarchical Attention Flow [32] models the interactions among passages, questions and candidate options to adequately leverage candidate options for multi-choice reading comprehension task.