# Machine Reading Comprehension Model Based on Fusion of Mixed Attention
## 4 Method and Models
### Mixed Attention Layer  
Due to the varying effects of different types of attention interaction mechanisms on the model's predictive ability, it is necessary to select a suitable attention mechanism that enables the model to learn deeper semantic features. This is inspired by the research of [PERSON] et al. [12], which found that self-attention mechanisms with a validated dot product to the Transformer model are greatly affected by learning attention through random alignment matrices and token-token interactions. Therefore, we propose two variants of self-attention mechanisms: Random Synthesizer and Dense Synthesizer. The conventional self-attention mechanism follows a standard computation method. It calculates the relevance scores between each token and other tokens, which yields a weight matrix \(A\). To obtain the final self-attention representation, the weight matrix \(A\) is normalized, and the corresponding key-value pairs are weighted and summed. The self-attention mechanism learns the connections between individual tokens and the overall  
Figure 1: BERT_hybrid model. sequence, allowing the model to capture both local and global relationships. The structure of the self-attention mechanism is illustrated in Figure 2.  
The dot-product-based self-attention mechanism has drawbacks, as it heavily relies on specific instances for computing self-attention through token-token interactions, incorporating alignment interaction information contained within the instance. This limits its generalization capability. Predicting answers using the dot-product-based self-attention mechanism is inherently unstable and sensitive to specific instances, compromising the model's ability to learn universal and generalizable features. To address this limitation, the Random Synthesizer attention mechanism was proposed to avoid an excessive focus on specific instance tokens and provide a certain degree of feature generalization capabilities. In the Random Synthesizer, the random combination attention is initialized with random values and jointly trained with the model. It exhibits a strong generalization capability because it does not depend on specific instance tokens. Moreover, the use of the same alignment pattern for each case ensures consistent and reliable outcomes. The Random Synthesizer primarily focuses on global attention; its structure is displayed in Figure 3.  
The Dense Synthesizer is used for learning local attention weights. It generates individual vectors by giving specific attention to the information carried  
Figure 3: Structure of Random Synthesizer. Figure 2: Structure of the self-attention mechanism. achieved by sequentially processing the input sequence and applying linear transformations to each dimension of the attention vector, resulting in a weight matrix \(R\), as depicted in Figure 4. The linear transformations are represented by Equation (5):  
\[Q_{N}=softmax(L_{N}(T_{i,N})) \tag{5}\]  
The Dense Synthesizer attention mechanism performs linear transformations on each row of tokens by using the transformation function \(\mathrm{L_{N}}\), which comprises a multilayer feedforward neural network. In this scenario, \(i\) represents the \(i\)-th token within \(T_{i,N}\), and \(N\) denotes the width of the matrix [13]. There are two self-attention mechanisms which differ in their underlying principles and implementation methods. As a result, distinct attention weight matrices are generated. In this layer, a hybrid approach is utilized to harness the advantages of both self-attention mechanisms. This enables simultaneous attention to local and global information and striking a balance between the two. By combining the two attention mechanisms, the hybrid attention fusion model avoids excessive reliance on specific input samples and efficiently extracts essential feature information from the original sequence; thus, the model's problem-solving efficiency is enhanced.