# Thangka Image--Text Matching Based on Adaptive Pooling Layer and Improved Transformer
###### Abstract  
Image-text matching is a research hotspot in the multimodal task of integrating image and text processing. In order to solve the difficult problem of associating image and text data in the multimodal knowledge graph of Thangka, we propose an image and text matching method based on the Visual Semantic Embedding (VSE) model. The method introduces an adaptive pooling layer to improve the feature extraction capability of semantic associations between Thangka images and texts. We also improved the traditional Transformer architecture by combining bidirectional residual concatenation and mask attention mechanisms to improve the stability of the matching process and the ability to extract semantic information. In addition, we designed a multi-granularity tag alignment module that maps global and local features of images and text into a common coding space, leveraging inter- and intra-modal semantic associations to improve image and text accuracy. Comparative experiments on the Thangka dataset show that our method achieves significant improvements compared to the VSE baseline method. Specifically, our method improves the recall by 9.4% and 10.5% for image-matching text and text-matching images, respectively. Furthermore, without any large-scale corpus pre-training, our method outperforms all models without pre-training and outperforms two out of four pre-trained models on the Flickr30k public dataset. Also, the execution efficiency of our model is an order of magnitude higher than that of the pre-trained models, which highlights the superior performance and efficiency of our model in the image-text matching task. Thangka; image-text matching; adaptive pooling layer; bidirectional residual connection; masking attention mechanisms +
Footnote †: journal:  
Embedding (VSE) [2] is a commonly used image-text matching method. It embeds images and texts for efficient cross-modal retrieval, and typically has the following steps. The image and text are extracted as features by separate visual and text encoders. These features are then projected into a joint embedding space and pooled to form fixed-length vectors. Then, the similarity calculation is utilized to measure the distance between instances and a suitable target is chosen for optimization. For example, [PERSON] [3] proposed the Stacked Cross Attention Network (SCAN), which utilizes attention mechanisms to process features of images and texts separately and compute potential relationships between corresponding regions, achieving favorable results. However, this model overlooks potential semantic relationships in the context. [PERSON] [4] introduced the Context-Aware Attention Network (CAAN), which enhances the focus on potential local semantics in the context and effectively aggregates contextual information, capturing deeper intra-modal correlations and improving retrieval performance. Similarly, to enhance attention to semantic relationships, [PERSON] [5] proposed the Iterative Matching with Recurrent Attention Memory (IMRAM) method, which iteratively updates cross-modal attention cores to explore fine-grained correspondences between images and texts. However, there are still problems, such as ignoring the relationship between text semantic features and insufficient acquisition of correlation between modalities. With the rise of large language models in the field of BERT [6] and NLP, various large-scale pre-training models have appeared in the field of CV, which has also promoted the development of visual-text cross-modal pre-training models. Their usual mode is, first, pre-training on some set image and text tasks, and then fine-tuning on downstream tasks. Thanks to the training of large-scale data, their performance on downstream tasks far exceeds that of non-pre-trained models, and they have very good results in image and text retrieval and matching. For example, [PERSON] [7] proposed a BERT architecture to learn the joint representation of images and text by integrating the feature extraction process, and achieved good results.