## 2 Methodology
### Cross-Attention Layer (CAL)  
This layer is used for attention convergence of region-level image features with text features as a clue. First, the correlation weights between each area and texts are determined. Next, a weighted summation is performed with these weights to generate the overall semantic features of images. Then, the image-text similarities are calculated. Finally, the matrix of image-text similarities is output. The detailed structure of the CAL is shown in Figure 4. Below is the concrete calculation process. Figure 3: The general process of region-level feature extraction from an image. The red squares passed through by the straight line at the last step represent the vector of image features within some area, whose dimension is 2048 in ResNet152. **Step 1:** Use the three fully connected layers to map region-level image features into a key tensor and a value tensor containing all semantic information of images and text features into a query matrix containing all semantic information of texts. \[M_{key}=Fc\_key(F,\theta_{key}) \tag{8}\]  
\[M_{val}=Fc\_val(F,\theta_{val}) \tag{9}\]  
\[M_{que}=Fc\_que(T^{\prime},\theta_{que}) \tag{10}\]  
At this point, \(M_{key}\) and \(M_{val}\) are both \(N\times 2048\times L\) tensors, where \(N\) represents the number of images, 2048 denotes the dimension of space, and \(j\) denotes the number of image areas. \(M_{que}\) is an \(N\times 2048\) matrix, where \(N\) represents the number of texts and 2048 denotes the dimension of space. **Step 2:** Calculate the cosine distance between each area \(k_{ij}\in\mathbb{R}^{2048}\) of each image \(k_{i}\) in \(M_{key}\) and each text \(q_{l},q_{l}\in\mathbb{R}^{2048}\) in \(M_{que}\). \[sim(k_{ij},q_{l})=\frac{k_{ij}^{T}q_{l}}{\|k_{ij}\|\|q_{l}\|} \tag{11}\]  
This formula expresses the cosine distance between the \(j\)th area of the \(i\)th image and the \(l\)th text. **Step 3:** Normalize the cosine distance found from (11) to give the weight of correlation between the jth area of the ith image and the lth text. \[\alpha_{ij}^{(l)}=\frac{e^{g(h)sim(k_{ij},q_{l})}}{\sum\limits_{j=1}^{l}e^{g( h)sim(k_{ij},q_{l})}} \tag{12}\]  
where \(g(h)\) is a temperature control function, which will be discussed later. According to the weight scores, one can find the feature representation of the \(i\)th image fused with the \(l\)th piece of textual information (hereinafter referred to as image fusion representation):  
Figure 4: Detailed structural diagram of CAL. CAL is used for cross-modal information fusion, with three fully connected layers as its components. The inputs of the Fc_Value Layer and Fc_Key Layer are regional-level image features, and the outputs are a value matrix and a key matrix, respectively; the input of the Fc_Query Layer is text features, and the output is a query matrix. Finally, this layer will converge on the query matrix as a clue to get the final image feature representation and calculate the similarity score between it and texts. The output of the entire CAL is a similarity matrix of image-text pairs. \[f_{i}^{(l)}=\sum_{j=1}^{J}\alpha_{ij}^{(l)}v_{ij} \tag{13}\]  
where \(\alpha_{ij}\) is the weight of correlation between the \(j\)th area of the \(i\)th image and the \(l\)th text as defined in (12) \(v_{ij}\) is the \(j\)th area of image \(v_{i}\) in \(M_{val}\). **Step 4:** Calculate the similarity matrix \(S\) between image features \(F^{\prime}\) and text features \(T^{\prime}\). The similarity between the \(i\)th image and the lth text can be formulated as:  
\[S_{il}=\frac{f_{i}^{(l)}T^{\prime}t_{j}^{\prime}}{\|f_{i}^{(l)}\|\|t_{j}^{ \prime}\|} \tag{14}\]  
**Step 5:** About the temperature control function \(g(h)\): Since the cosine distances range within \([-1,1]\), there will be no obvious distinction in the Wright score \(a\) between different areas. Therefore, it would be best to multiply it by an amplification coefficient.