# Neural Machine Reading Comprehension: Methods and Trends
## 3 Deep-Learning-Based Methods
### Typical Deep-Learning Methods
#### 3.2.2 Feature Extraction
As shown in Figure 9, given word embeddings of a question \(x_{q}\in\mathbb{R}^{|I|\times d}\), where \(|I|\) represents the length of the question and \(d\) denotes the dimension of word embeddings, the convolution layer has two filter sizes \(f_{t}\times d(\forall t=2,3)\) with \(k\) output channels (\(k=2\) in the example in Figure 9). Each filter produces a feature map of shape \((|I|-t+1)\times k\) (padding is 0 and stride is 1), which are pooled to generate a k-dimensional vector. The two k-dimensional vectors are concatenated to form a 2k-dimensional vector, represented by \(Q\). Although both n-gram models and CNNs can focus on local features of the sentence, training parameters in n-gram models increase exponentially with larger vocabularies. By contrast, CNNs can extract local information in a more compact and efficient way regardless of the vocabulary size, as there is no need for them to represent every n-gram in the vocabulary. In addition, CNNs can be trained in parallel, which is faster than RNNs. One major shortcoming of CNNs is that they can extract only local information but are not capable of dealing with long sequence. (3) Transformer  
The transformer, proposed by [PERSON] et al. [41] in 2017, is a powerful neural network model that has shown promising performance on various NLP tasks [31; 32]. In contrast to RNN-based or CNN-based models, the transformer is mainly based on the attention mechanism with neither recurrence nor convolution. Owing to multi-head self-attention, this simple architecture not only excels in alignment but also can be run in parallel. Compared to RNNs, the transformer requires less time to train, while it pays more attention to global dependencies, in contrast to CNNs. However, without recurrence and convolution, the model cannot make use of the order of the sequence. To incorporate positional information, [PERSON] et al. add position encoding computed by sine and cosine functions. The sum of positional and word embeddings is fed to the transformer as inputs. Figure 10 presents the simple architecture of the transformer. In practice, models usually stack several blocks with multi-head self-attention and feed-forward network. Figure 9: Using CNNs to extract features of question. QANet, introduced by [PERSON] et al. [25], is a representative MRC model that uses the transformer. The basic encoder block of QANet is a novel architecture, which combines the multi-head self-attention defined in the transformer with convolutions. The experimental results show that QANet achieves the same accuracy on SQuAD as the prevalent recurrent models with much faster training and inference speed. In general, most MRC systems use RNNs in the feature extraction module because of their superiority in handling sequential information. To accelerate the training process, some researchers substitute RNNs with CNNs or the transformer. CNNs are highly parallelized and can obtain rich local information with feature maps in different sizes. The transformer can mitigate the side-effect of long dependency and improve computational efficiency.