# An Idiom Reading Comprehension Model Based on Multi-Granularity Reasoning and Paraphrase Expansion
## 3 Model
### BERT-IDM\({}_{\text{base}}\) Model
This attention masking matrix is a diagonal matrix with 1 s on the main diagonal and 0 s elsewhere. Its role is to mask the matrix elements on the main diagonal, thereby shielding the self-information of each vector. This can ensure that when calculating attention weights, each vector only focuses on the information of other vectors, reducing computational complexity and improving the efficiency of the model. At the same time, the introduction of the attention masking matrix can also further ensure the security of the model and avoid interference from the self-information of each vector on the performance of the model. The attention mask pattern is shown in Figure 3. Based on this, the compatibility of the model with multi-layer deep stacking architectures and residual connections was restored by setting up an isolation mechanism for vectors. Figure 2: Diagram of pretraining process of BERT-IDM. The key to Transformer, the core structure of BERT, is to use the self-attentive mechanism to compute the \(Attention(Q,K,V)\) vector. Equivalent \(Q\), \(K\), and \(V\) vectors were selected in the self-attentive mechanism of BERT. If the original BERT's self-attention layer-wise propagation algorithm is used to select the \(K\) and \(V\) vectors in subsequent hidden layers, the content information in the token vectors will diffuse into the query vector \(Q\) in the subsequent layers, because the \(K\) and \(V\) vectors have already been fused in the upper layers. The final vector used for the prediction output is the query vector \(Q\), so if this problem is not addressed, the SMSA mechanism will fail after the first layer of input in a deep model. To solve this problem, we set up an isolation mechanism for the \(K\) and \(V\) vectors to restore the compatibility of the model with a multi-layer deep stacking architecture and residual connections. Specifically, all \(K\) and \(V\) vectors in the encoding layers were fixed to a constant value, and only the query vector \(Q\), which was used for the final prediction output and did not contain self-content information, was updated across layers. The fixed values of the \(K\) and \(V\) vectors were determined by the combination of the input embedding sequence \(E\) and the position vector \(P\). The modified SMSA updated the attention calculation process as shown in Figure 4. Algorithm 1 formatively defines the attention flow isolation of SMSA and describes the flow in detail. This algorithm simultaneously reduced the time complexity of the large number of linear operations in the computation of the self-attentive mechanism by fixing the vector. Figure 4: Calculation process of SMSA mechanism. Figure 3: Schematic Diagram of SMSA mechanism. ```
Input: Query vector \(Q\), key vector \(K\), value vector \(V\) Output: Query vector for predicting output \(Q^{\prime}\)
1 The sequence of participle embeddings \(E\) and the corresponding position vector \(P\) are loaded;
2 The query vector \(Q_{1}\) of the first layer input is initialized as the position vector \(P\), and the key vector \(K\) and value vector \(V\) are fixed as the vector \(E+P\) and not updated in subsequent layers;
3 The input \(Q_{m}\) and \(K\) vectors in the \(m\) layer pass through the attention mask module after matrix multiplication and deflation operations; proceed through the Softmax layer; and are then subject to matrix multiplication with the vector \(V\) before outputting the attention module to obtain the intermediate output \(T_{m}=Q_{m}+1\) in accordance with Equation (1): \[T_{m}=SMSA(Q_{m},K,V)=Softmax(AttnMask(\frac{Q_{m}K^{T}}{\sqrt{d}}))V;\] (1)
4 The output of this layer is obtained by forward propagation in the fully connected layer (FFN), denoted as \(S_{m}\). \(S_{m}\) satisfies Equation (2) after residual concatenation and regularization based on the results of step 3: \[S_{m}=f(Norm(Add(Q_{m},T_{m}))),\] (2) where \(f(.