# Neural Machine Reading Comprehension: Methods and Trends
## 3 Deep-Learning-Based Methods
### Typical Deep-Learning Methods
#### 3.2.2 Feature Extraction  
The feature extraction module is often placed after the embedding layer to extract features of the context and question separately. It further pays attention to mining contextual information at the sentence level based on various types of syntactic and linguistic information encoded by the embedding module. Recurrent neural networks (RNNs), convolution neural networks (CNNs) and Transformer architecture are applied in this module, and we will give an illustration in detail in this part. (1) Recurrent Neural NetworksRNNs are popular models that have shown great promise for dealing with sequential information. RNNs are called _recurrent_ as outputs in each time step depending on the previous computations. RNN-based models have been widely used in various NLP tasks, such as machine translation, sequence tagging and question answering. In particular, long short-term memory (LSTM) [47] networks and gated recurrent units (GRUs) [38], variants of RNNs, are much better at capturing long-term dependencies than plain ones are and can alleviate gradient explosion and vanishing problems. Since the preceding and following words have the same importance in understanding the given word, many researchers use bidirectional RNNs to encode the context and question embeddings in MRC systems. The context and question embeddings are denoted as \(x_{p}\) and \(x_{q}\), respectively, and we will illustrate how the feature extraction module with bidirectional RNNs handles those embeddings and extracts sequential information. In terms of questions, the feature extraction process with bidirectional RNNs can be sorted into two types: word-level and sentence-level. In word-level encoding, feature extraction outputs for each question embedding \(x_{qj}\) at time step \(j\) can be denoted as follows:  
\[Q_{j}=[\overrightarrow{RNN}(x_{qj});\overleftarrow{RNN}(x_{qj})], \tag{1}\]  
where \(\overrightarrow{RNN}(q_{xj})\) and \(\overleftarrow{RNN}(q_{xj})\) denotes forward and backward hidden states of bidirectional RNNs, respectively. This process is shown in Figure 7 in detail. Figure 7: Word-level encoding for questions. By contrast, sentence-level encoding regards the question sentence as a whole. The feature extraction process can be denoted as:  
\[Q=[\overrightarrow{RNN}(x_{q||i});\overleftarrow{RNN}(x_{q0})], \tag{2}\]  
where \(|I|\) is the length of the question, \(\overrightarrow{RNN}(x_{q||})\) and \(\overleftarrow{RNN}(x_{q0})\) represent final forward and backward outputs of RNNs, respectively. To be more concrete, we demonstrate this sentence-level encoding process in Figure 8.  
As the context in an MRC task is usually a long sequence, researchers use word-level feature extraction to encode sequential information of context. Similar to question encoding, the feature extraction process with bidirectional RNNs for context embedding \(x_{pi}\) at time step \(i\) can be denoted as:  
\[P_{i}=[\overrightarrow{RNN}(x_{pi});\overleftarrow{RNN}(x_{pi})]. \tag{3}\]  
Although RNNs are capable of modeling sequential information, their training process is time-consuming as they cannot be processed in parallel. (2) Convolution Neural Networks  
CNNs are widely used in computer vision. When applied to NLP tasks, one-dimensional CNNs show their superiority in mining local contextual information with sliding windows. In CNNs, each convolutional layer applies a different scale of feature maps to extract local features in diverse window sizes. The outputs are then fed to pooling layers to reduce dimensionality but keep the most significant information to the greatest extent. Maximum and average operations on the results of each filter are common ways to do pooling. Figure 9 shows how the feature extraction module uses CNNs to mine the local contextual information of a question. Figure 8: Sentence-level encoding for questions.