# Thangka Image--Text Matching Based on Adaptive Pooling Layer and Improved Transformer
## 2 Thangka Image-Text Matching Model
### Multigranularity Alignment Module  
With the above processing, we obtain a series of multi-granular features for images and text, which next need to be modal aligned. The main job of the alignment module is to map the input samples into a shared embedding space so that the similarity between them can be compared; the module performs this by learning a mapping function that converts the input samples into a low-dimensional vector representation. This has the advantage that it allows different types of inputs (e.g., text, images, or other data) to be represented uniformly as vectors, thus facilitating the computation of similarity between them. In matching problems, the calculation of similarity between positive and negative samples is critical. Positive samples are pairs of samples that belong to the same category or have similar attributes, while negative samples are pairs of samples that belong to different categories or have dissimilar attributes. By calculating the similarity between positive and negative samples, we can quantify the degree of difference between them. The purpose of this is to make the similarity between the positive samples as high as possible and the similarity between the negative samples as low as possible during the training process. In this way, we can train the model to learn the ability to distinguish between positive and negative samples, leading to better matching performance. Based on the positive and negative samples' feature information, we construct four types of matching relationships: matching between the global region of the image and the text sentence, matching between the target region of the image and the text sentence, matching between the text entity and the global region of the image, and matching between the target region of the image and the text entity. We calculate different matching scores and ensure that the matching score of the final positive image-text pair is greater than the matching score of the negative image-text pair. The calculation formulas are shown in Equations (9)-(11):  
\[L_{g1}=T\bigg{(}X_{i}^{I},X_{i}^{T},X_{j}^{T}\bigg{)}+T\bigg{(}X_{i}^{T},X_{i} ^{I},X_{j}^{I}\bigg{)} \tag{9}\]  
\[L_{l1}=T\bigg{(}X_{i}^{I},x_{i}^{T},x_{j}^{T}\bigg{)}+T\bigg{(}X_{i}^{T},x_{i} ^{I},x_{j}^{I}\bigg{)} \tag{10}\]  
\[L_{l2}=T\bigg{(}X_{i}^{I},x_{i}^{T},x_{j}^{T}\bigg{)}+T\bigg{(}x_{i}^{T},x_{i} ^{I},x_{j}^{I}\bigg{)} \tag{11}\]  
In these equations, \(X_{i}^{I}\), \(X_{i}^{T}\), \(X_{j}^{T}\) represent the image output sample, text positive sample, and text negative sample in the positive image-text pair, respectively. The subscripts\(i\) and \(j\) represent the data in the samples, and I and T correspond to the image and text modalities, respectively. \(L_{g1}\) is used to calculate the matching relationship in (1), \(L_{l1}\) is used to calculate the matching relationships in (2) and (3), and \(L_{l2}\) is used to calculate the matching relationship in (4). The function T represents the triplet loss function \(T(u,P,N)\)[16], and its formula is shown in Equation (12):  
\[T(u,P,N)=\max(\frac{1}{p}{\sum}_{p\in P}\,Sim(u,p)+\frac{1}{N}{\sum}_{m\in N} \,Sim(u,n)+d,0) \tag{12}\]  
In this paper, a set of image-text pairs (\(I_{i},E_{i}\)) is established, which includes negative samples of text and negative samples of images. \(I_{i}\) represents the positive image sample, and \(E_{i}\) represents the positive text sample. In the loss function \(T(u,P,N)\), \(u\) represents the training prediction sample, \(P\) represents the positive samples, and \(N\) represents the negative samples. The scalar \(d\) is used to control the distance range between the output sample \(u\) and the positive and negative samples \(P\) and \(N\). \(Sim\) is the similarity function that calculates the similarity between the output sample and the positive/negative samples. In this paper, the cosine distance similarity function is used. The max() function is used to set the result to zero whenever the computed value of the function is less than or equal to zero.