# Neural Machine Reading Comprehension: Methods and Trends
## 3 Deep-Learning-Based Methods
### Typical Deep-Learning Methods
#### 3.2.1 Embeddings
In particular, for MRC tasks, BERT is so competitive that using it with a simple answer prediction approach shows promise. Despite its outstanding performance, BERT's pre-training process is time and resource-consuming which makes it nearly impossible to pre-train without abundant computational resources. (3) Multiple Granularity  
Word-level embeddings pre-trained by Word2 Vec or GloVe cannot encode rich syntactic and linguistic information, such as part-of-speech, affixes, and grammar, which may not be sufficient for deep machine understanding. To incorporate fine-grained semantic information into word representations, some researchers have introduced approaches to encode the context and the question at different levels of granularity. * Character Embeddings  
Character embeddings represent words at the character level. Compared to word-level representations, they are not only more suitable for modeling sub-word morphologies but also can alleviate the out-of-vocabulary (OOV) problem. [PERSON] et al. [16] add character-level embeddings to their Bi-DAF model for the MRC task. They use CNNs to obtain character-level embeddings. Each character in the word is embedded into a fixed-dimension vector, which is fed to CNNs as 1D inputs. After max-pooling the entire width, the outputs of CNNs are embeddings at the character level. The concatenation of word-level and character-level embeddings are then fed to the next module as input. In addition, character embeddings can be encoded with bidirectional LSTMs [26; 43]. For each word, the outputs of the last hidden state are considered to be its character-level representation. Moreover, word-level and character-level embeddings can be combined dynamically with a fine-grained gating mechanism rather than simple concatenation to mitigate the imbalance between frequent and infrequent words [44]. * Part-of-Speech Tags  
A part-of-speech (POS) is a particular grammatical class of words, such as nouns, adjectives, or verb. Labeling POS tags in NLP tasks can illustrate complex characteristics of word use and in turn contribute to disambiguation. To translate POS tags into fixed-length vectors, they are regarded as variables, randomly initialized in the beginning and updated while training. * Name-Entity Tags  
Name entity, a concept in information retrieval, refers to a real-world object, such as a person, location, or organizations, with a proper name. When asking about such objects, name entities are probable answer candidates. Thus, embedding name-entity tags of context words can improve the accuracy of answer prediction. The method of encoding name-entity tags is similar to that of POS tags mentioned above. * Binary Feature of Exact Match (EM)  
This feature, which measures whether a context word is in the question, was first used in the conventional entity-centric model proposed by [PERSON] et al. [45]. Later, some researchers used it in the Embedding module to enrich word representations. The value of this binary is 1 if a context word can be exactly matched to one word in the query, otherwise its value is 0. More loosely, [PERSON] et al. [27] use partial matching to measure the correlation between context words and question words. For instance, \"teacher\" can be partially matched with \"teach\". * Query-Category  
The types of questions (what, where, who, when, why, how) can usually provide clues to search for the answer. For instance, a \"where\" question pays more attention to spatial information. [PERSON] et al. [46] introduce a method to model different question categories in end-to-end training. They first obtain query types by counting the key word frequency. Then the question type information is encoded to one-hot vectors and stored in a table. For each query, they look up the table and use a feed-forward neural network for projection. The query-category embeddings are often added to the query word embeddings. The embeddings introduced above can be combined freely in the embedding module. [PERSON] et al.