# Thangka Image--Text Matching Based on Adaptive Pooling Layer and Improved Transformer
## 2 Thangka Image-Text Matching Model  
The proposed framework of the Thangka image-text matching model in this paper is shown in Figure 1. It consists of four main components: the image feature extraction module, the text feature extraction module, the encoder module, and the multi-granularity alignment module.  
For the imaging modality, considering the diverse target objects in Thangka images, the small pixel blocks occupied by each object, and the existence of pixel adhesion among different targets, we first use the top-down visual attention mechanism based on the Faster R-CNN [10] model to extract the target regions in Thangka images. Then, in the process of abstracting the feature matrix into feature vectors, adaptive pooling techniques are introduced, allowing the model to automatically learn the weight determination process. The subsequent visual and textual encoder modules learn and combine these features to generate fixed-size embedding features, which better extract the local feature vectors of the target objects in Thangka images and are represented as \(E_{u}^{I}:\{i_{1},i_{2},i_{3},\ldots,i_{u}\};i_{u}\) represents the \(u\)-th feature dimension of the vector. The global features are extracted using the feature extraction network of the trained Faster R-CNN, represented as \(C^{I}\).  
For the text modality, the given Thangka text is first converted into a vector representation. A pre-trained entity relation extraction model is used to extract (entity-relation-entity) triplets from the Thangka text data, which are then merged into text. The merged text and entity fragments are separately processed through a pre-trained bidirectional GRU [11] model, which introduces adaptive pooling layers to extract text features while preserving contextual semantic information as much as possible. This process yields local text feature vectors containing contextual semantic structures, represented as \(E_{u}^{T}:\{t_{1},t_{2},t_{3},\ldots,t_{u}\}\). The global feature vector for the text is extracted using the feature extraction module, represented as \(C^{T}\).  
After extracting the global and local features from both the image and text modalities, we use the encoder module to establish multi-granularity semantic associations between the two modalities. The encoder adopts an improved Transformer network  
Figure 1: Model Framework Diagram.  
structure that combines bidirectional residuals and masked attention mechanisms. This structure solves potential gradient vanishing and model collapse issues and focuses on learning multi-granularity semantic information with similar structures between modalities. It outputs multi-granularity feature representations that combine information from different dimensions.  
Finally, based on the obtained multi-granularity feature vectors, the modal alignment module calculates the losses for minimizing the correlation of local corresponding region features between positive samples, the correlation between regions and the whole, and maximizing the correlation of local corresponding region features between negative samples. This ensures that the learned region features have good discriminability, thereby guaranteeing the performance of Thangka image-text matching.