# Neural Machine Reading Comprehension: Methods and Trends
## 3 Deep-Learning-Based Methods
### Typical Deep-Learning Methods
#### 3.2.1 Embeddings
concatenate the outputs of an MT encoder with word embeddings pre-trained by GloVe to represent the context and question and feed them through the coattention and dynamic decoder implemented in a dynamic coattention network (DCN) [17]. DCN with CoVe outperforms the original DCN on the SQuAD dataset, which illustrates the contribution of contextualized word representations to downstream tasks. However, pre-training CoVE requires a large parallel corpus. Its performance will degrade if the training corpus is not adequate. * ELMo  
Embeddings from language models (ELMo), proposed by [PERSON] et al. [30], is another type of contextualized word representation. To get ELMo embeddings, they first pre-train a bidirectional language model (biLM) with a large text corpus. Compared to CoVe, ELMo breaks the constraint of limited parallel corpus and obtains richer word representations by collapsing outputs of all biLM layers into a single vector with a task-specific weighting rather than using outputs of the top layer. Model evaluations illustrate that different levels of LSTM states can capture diverse syntactic and linguistic information. When applying ELMo embeddings to MRC models, [PERSON] et al. choose an improved version of bidirectional attention flow (Bi-DAF), introduced by [PERSON] and [PERSON] [40], as the baseline and improved the state-of-the-art single model by 1.4% on the SQuAD dataset. ELMo, which can be easily integrated with existing models, shows promising performance on various NLP tasks, but it is limited by the insufficient feature extraction capability of LSTM. * GPT  
Generative pre-training (GPT) [31], is a semi-supervised approach combining unsupervised pre-training and supervised fine-tuning. Representations pre-trained by this method can transfer to various NLP tasks with little adaptation. The basic component of GPT is a multi-layer transformer [41] decoder that mainly uses multi-head self-attention to train the language model and allow longer semantic structure to be captured compared to RNN-based models. After training, the pre-trained parameters are fine-tuned for specific downstream tasks. In terms of MRC problems such as multiple choice, [PERSON] et al. concatenate the context and the question with each possible answer and process such sequences with transformer networks. Finally, they produce an output distribution over possible answers to predict correct answers. GPT achieves an improvement of 5.7% on the RACE [10] dataset compared with the state-of-the-art. Seeing the benefit brought by contextualized word representations pre-trained on large-scale datasets, [PERSON] et al. [42] later propose GPT-2, which is pre-trained on a larger corpus, WebText, with more than 1.5 billion parameters. Compared to the previous model, the layers of transformer architecture are increased from 12 to 48. Moreover, single task training is substituted with a multi-task learning framework, which makes GPT-2 more generative. This improved version shows competitive performance even in zero-shot setting. However, the transformer architecture used in GPT and GPT-2 is unidirectional (left-to-right), which cannot incorporate context from both directions. This may be the major shortcoming and limits its performance on downstream tasks. * BERT  
Considering the limitations of unidirectional architecture applied in previous pre-training models such as GPT, [PERSON] et al. [32] propose a new model, bidirectional encoder representation from transformers (BERT). With the masked language model (MLM) and next-sentence prediction task, BERT can pre-train deep contextualized representations with a bidirectional transformer, encoding both left and right context for word representations. As transformer architecture cannot extract sequential information, [PERSON] et al. add positional embeddings to encode position. Owing to the bidirectional language model and transformer architecture, BERT outperforms state-of-the-art models on 11 NLP tasks.