# Image-Text Matching Model Based on CLIP Bimodal Encoding
## 3 Joint Encoding Model for Image-Text Matching
### Image Encoder  
In recent years, due to the rapid development of transformers in the field of natural language processing [17], many scholars and experts have been applying this model to the computer vision domain. For image data, this paper utilizes the Vision Transformer for encoding, which consists of three components: Patch Embedding, Transformer Encoder, and a multi-layer perceptron classifier. In a standard transformer, the required input is a token vector, i.e., a two-dimensional matrix. However, the standard image input data format is [H, W, C], which evidently does not comply with the input requirements of the transformer. Therefore, the first step of the model in this paper employs two operations, Patch and Embedding, to transform the image data into a two-dimensional matrix. The Patch operation is implemented using convolutional operations. For instance, for an image with dimensions [224; 224; 3], representing a 3-channel 224 \(\times\) 224 image, 768 16 \(\times\) 16 convolutional kernels are used with a stride of 16 and zero padding. According to the convolutional size change formula:  
\[N=\frac{W-F+2P}{S}+1 \tag{1}\]  
where \(W\) is image size, \(F\) is convolution kernel size, \(P\) is edge filling size, \(S\) is step size. By calculation, a feature map of [14; 14; 768] can be obtained. After the Flatten operation, a two-dimensional matrix of [196; 768] can be obtained. In order for the model to acquire the  
Figure 1: Image–Text Matching Architecture. category information of the current image, a class token is added at the beginning of the input sequence in Patch Embedding to represent the category information of the current image. Therefore, the overall dimension becomes [197; 768]. As the transformer is not as adept at handling sequential data as RNN, each component in the token needs to be supplemented with positional encoding information. The approach is to directly stack the positional encoding vector onto the tokens. Thus, after processing by the Embedding layer, the final input dimension of the image data is [197; 768]. This format meets the input requirements of the Transformer Encoder. The structure of the Transformer Encoder consists of stacking multiple layers of Encoder Blocks. Each Encoder Block includes Layer Norm, a multi-head attention mechanism, Dropout, and an MLP Block. The structure of the Encoder Block is shown in Figure 2.  
In contrast to the word vector approach in natural language processing, the Encoder Block first conducts layer normalization on the incoming tokens vector. Layer normalization is a common data preprocessing transformation method, which ensures that all data conforms to a standard normal distribution with a mean of 0 and a variance of 1. This can accelerate the model training speed, enhance model robustness, and contribute to achieving better model performance. The multi-head attention module aims to enable the model to comprehend data from multiple dimensions, and the computation in multi-head attention is parallel, aligning with modern hardware architecture to expedite computational efficiency. The self-attention mechanism used in multi-head attention allows the output vector to consider all preceding input vectors. After the multi-head self-attention layer and layer normalization, an output vector will be obtained: \(O=[o_{1},o_{2},\dots,o_{m}]\in R^{m\times d}\), The calculation formula is as follows:  
\[O=LayerNorm(Z+MultiHead(Z)) \tag{2}\]  
Figure 2: Encoder Block Structure Diagram. Then use a feedforward neural network to perform operations on each component in \(O\), as follows:  
\[y_{i}=LayerNorm(o_{i}+FFN(o_{i})) \tag{3}\]  
By performing the above operation on all m regions, we obtain \(Y=\{y_{1},y_{2},\dots,y_{m}\}\), we then use the average pooling operation to obtain the final embedding vector:  
\[q=\frac{1}{m}\sum_{i=1}^{m}y_{i} \tag{4}\]  
In order to prevent overfitting, this paper uses Dropout operation, whose dimensions are invariant after processing.