## 3 Proposed Method
### Repeat Question Module  
Similar to the ViT [41] model, we consider a Transformer encoder as a Transformer layer. The repeat question learning module consists of \(M_{depth}\) similar layers with residual connections (Figure 3). In the attention part, we utilize nonlinear scaled dot-product attention. The attention score input comprises the key and value from the input. We scale each attention score by the input dimension \(M_{embed}\), with the scaling factor being \(\frac{1}{\sqrt{M_{embed}}}\). Here, a Tanh activation function is applied to project the attention scores into the nonlinear range of \((-1,1)\), with the aim of fitting the nonlinear sequence space and separate positive and negative feature weights. This approach helps in reducing the amplification effect of scores for ineffective blocks on the total weight when the input dimension is large, assigning higher weights to effective blocks in the Softmax classification output. The attention score and attention algorithm are shown as follows:  
\[AttentionScore(Q,K)=Tanh\Big{(}Q\times K^{T}\Big{)}, \tag{5}\]  
\[Attention\ (Q,K,V)=Softmax\Big{(}\frac{AttentionScore(Q,K)}{\sqrt[3]{M_{ embed}}}\Big{)}.V. \tag{6}\]  
In the first and second layers, an auxiliary masking mechanism is applied for the attention weights of the multi-head attention mechanism. This mechanism helps each attention head to focus on text sequence blocks with high attention scores while minimizing weight waste on ineffective words as much as possible. After obtaining \(Q\times K^{T}\), this mechanism masks elements below the threshold \(f=0.5\) in the attention score matrix \(AttentionScore\). Elements with values lower than the threshold \(f=0.5\) are masked to \(X_{min}=10^{-9}\):  
\[\forall x_{i,j}\in AttentionScore,\ x_{i,j}=\left\{ \begin{matrix}x_{i,j}&x_{i, j}\geq f\\ 10^{-9}&else\end{matrix} \right.. \tag{7}\]In the subsequent layers, this masking mechanism is removed. The encoder network layers can fully learn from the input while passing a copy of the input to the next layer. The predictions of the current layer are additively fused with the input copy of this layer. In this work, \(M_{embed}\) is projected to dimensions \(Query_{h}\), \(Key_{h}\), and \(Value_{h}\), serving as the input for h attention learners. Nonlinear dot-product scaling attention is applied to each attention learner, allowing the model to focus on feature information in various subspaces of the sequence samples. The granularity of feature extraction is finer compared with a single self-attention mechanism. This module employs a multi-head attention mechanism with h attention heads, where \(Query_{h}=Key_{h}=Value_{h}=M_{embed}/h\), requiring the word embedding dimension from Section 3.1 to be evenly distributed across h subspaces. The joint feature learning matrix, denoted as \(W_{joint}\in R^{M_{embed}\times M_{padding}}\), is utilized for this purpose. The computation process of the multi-head attention mechanism is as follows. Here, we refer to the output of this part as \(Att_{out}\). \[ \begin{split} MultiHeadAttention(Q,K,V)=W_{joint}^{\text{T}}( concat[head_{1},head_{2},\dots,head_{h}])\\ where\ head_{i}=Attention(Query_{h},Key_{h},Value_{h})_{i}.\end{split}  \tag{8}\]  
After the attention network, a scaled MLP with a residual connection to the attention input is introduced. The scaling perceptron maps the output of the attention network to a high-dimensional space of dimension \(cls_{dim}\) and performs the first linear partitioning. Thereafter, the dimension of the matrix mapped by the first-dimension increment is flipped and mapped back to the original dimension \(M_{embed}\), with a ReLU activation function inserted in the middle of the scaling perception for linear feature correction. The presence of bias terms in the linear mapping operation affects the feature space distribution of the information. Accordingly, a layer normalization mechanism is established to address this issue.