## 3 Method
### Key Information Extraction
#### 3.1.2 Multimodal Encoder with Attention Refinement  
The multimodal encoder with attention refinement is built on the foundation of the Transformer encoder. Internally, it mainly consists of two parts: the multi-attention refinement module and a two-layer feed-forward neural network. Both parts incorporate residual connectivity and a layer normalization operation is performed after residual connectivity. The primary focus of this section is the multi-attention refinement module, as visualized in Figure 4a. General self-attention [9] starts with the input \(X=\{x_{1},x_{2},\ldots,x_{L}\}\in L\times d_{model}\) multiplied by three different matrices to obtain \(Q\) (the query used to match other units), \(K\) (the key used to be matched by other units), and \(V\) (the value needed to be extracted), i.e.,  
\[Q=XW^{Q},\ \ \ K=XW^{K},\ \ \ V=XW^{V} \tag{3}\]  
where \(W_{Q},W_{K}\in d_{model}\times d_{k},W_{V}\in d_{model}\times d_{v}\) are the trainable parameter matrices, \(L\) is the length of the input sequence, \(d_{model}\) is the model depth, and \(d_{k}\) is the depth of each attention head. Subsequently, the attention score \(A\) for each input cell is calculated as defined by Equation (4):  
\[A=softmax(\alpha)=softmax(\frac{Q\cdot K^{T}}{\sqrt{d_{k}}}) \tag{4}\]  
where \(d_{k}\) is set to avoid vanishing gradients, and \(\alpha\) is the similarity matrix between the input cells. In this paper, an attention refinement module is proposed to optimize the attention output of the Transformer encoder, as shown in Figure 4b. Specifically, first, the attention score \(A\) in Equation (4) is used as the input to the attention refinement module, and then all the attention scores before the current position \(t\in[0,L)\) are accumulated to obtain the attention score, \(\alpha_{t}\), at position \(t\). Next, the attention scores go through the convolutional neural network, and a 1\(\times\)1 convolution is used for feature dimensionality reduction. The refinement matrix is obtained by batch normalization and dimension adjustment. The whole process can be expressed as in Equations (5) and (6):  
Figure 4: **(a)** is a single self-attention refinement layer; (**b**) is the attention refinement module; (**c**) is the overall architecture of the multi-attention refinement module. The multi-attention refinement module is made up of several self-attention refinement layers. Each layer is created by adding a refinement module to the traditional Attention mechanism. \[\alpha_{t}=\sum_{k=1}^{t-1}A_{k} \tag{5}\]  
and  
\[R=reshape(norm(\max(0,conv(reshape(\alpha_{t}))))W^{R}) \tag{6}\]  
where the convolutional kernel size represents \(k=5\), \(W_{R}\in d_{c}\times n\) is the matrix of trainable parameters, \(d_{c}\) is the intermediate dimension of the convolutional layer, and \(n\) denotes the number of attention heads. The final step subtracts the refinement term, \(R\), from the initially generated similarity matrix, \(\alpha\), to obtain the final similarity matrix, \(B\)  
\[B=\alpha-R \tag{7}\]  
The \(R\) matrix can extract local coverage features to record the edges of the attended region and identify the incoming unattended region. The adjusted attentional similarity matrix \(B\) puts more attention on the unattended region of the input features. The \(i\)-th attention head \(H_{i}\) can be expressed as in Equation (8):  
\[H_{i}=softmax(B)V_{i} \tag{8}\]  
The final output of the multi-attention refinement module is a splicing of n attention heads followed by a linear transformation. Detailed network details are displayed in Figure 4c, i.e.,  
\[H=concat(H_{1},H_{2},\dots,H_{n})W^{n} \tag{9}\]  
where \(W^{n}\in nd\times d_{model}\) is the matrix of trainable parameters.