# Neural Machine Reading Comprehension: Methods and Trends
## 3 Deep-Learning-Based Methods
### Typical Deep-Learning Methods
#### 3.2.1 Embeddings  
The embedding module is an essential part of an MRC systems and is usually placed at the beginning to encode input natural language words into fixed-length vectors, which the machine can understand and deal with. As [PERSON] et al. [33] point out, minor choices made in word representation can lead to substantial differences in the final performance of the reader. How to sufficiently encode the context and question is the pivotal task of this module. In existing MRC models, word representation methods can be sorted into conventional word representation and pre-trained  
contextualized representation. To encode more abundant semantic and linguistic information, multiple granularity, which fuses word-level embeddings with character-level embeddings, part-of-speech, name entity, word frequency, question category, and so on, is also applied to some MRC systems. In the following parts, we will give a detailed illustration. (1) Conventional Word Representation  
* One-Hot  
This method [34] represents words with binary vectors, and its size is same as the number of words in the dictionary. In such vectors, one position is 1, corresponding to the word, while all others are 0. As a word representation approach at the early stage, it can encode words when the size of the vocabulary is not very large. However, this representation is sparse and may suffer from the curse of dimensionality with increased vocabulary. In addition, one-hot encoding cannot represent relationships among words. For instance, \"apple\" and \"pear\" belong to the category \"fruit\", but their word representations embedded by one-hot cannot show such a relationship. * Distributed Word Representation  
To address the shortcomings of representations such as one-hot, [PERSON] et al. [35] propose distributed word representation, which encodes words into continuous low-dimensional vectors. Closely related words encoded by these methods are not far from each other in vector space, which reveals the correlation of words. Various techniques to generate distributed word representations have been introduced, among which the most popular are Word2 Vec [36] and GloVe [37]. Besides successful applications in a variety of Natural Language Processing (NLP) tasks, such as machine translation [38] and sentiment analysis [39], vectors produced by these methods are also applied to many MRC systems. (2) Pre-trained Contextualized Word Representation  
Although distributed word representation can encode words in low-dimensional space and reflect correlations between words, it cannot efficiently mine contextual information. To be specific, a vector produced by the distributed word representation of one word is constant regardless of different contexts. To address this problem, researchers have introduced contextualized word representations, which are pre-trained with large corpora in advance and then directly used as conventional word representations or fine-tuned according to the specific tasks. This is a kind of transfer learning and has shown promising performance in a wide range of NLP tasks including machine reading comprehension. Even a simple neural network model can perform well in answer prediction with such a pre-trained word representation approach. * CoVE  
Inspired by successful cases in computer vision, which transfer CNNs pre-trained on a large supervised training corpus such as ImageNet to other tasks, [PERSON] et al. [24] try to bring the benefit of transfer learning to NLP tasks. They first train long short-term memory (LSTM) encoders of sequence-to-sequence models on a large-scale English-to-German translation dataset and then transfer the outputs of the encoder to other NLP tasks. As machine translation (MT) requires the model to encode words in context, the output of the encoder can be regarded as context vectors (CoVE). To deal with MRC problems, [PERSON] et al.