# Image-Text Matching Model Based on CLIP Bimodal Encoding
## 3 Joint Encoding Model for Image-Text Matching
### Text Encoder  
The sequential nature is an inherent characteristic of textual data, and the RNN model is specifically designed to handle sequential data. The RNN model possesses a \"memory\" function, enabling it to record information from previous time steps and pass it on for current time step processing. However, the drawback of the RNN model is that when the sequence is too long, the information from earlier distant time steps has minimal impact on the current time step, leading to the phenomenon of vanishing gradients. This is not very favorable for long sequential text. Subsequently, the Transformer model gained widespread usage. In 2018, [PERSON] et al. (2018) proposed the Bert model, whose underlying architecture is also based on the Transformer model. In simple terms, Bert is formed by stacking multiple transformer encoders. The structure of Bert model is shown in Figure 3.  
In text generation tasks, the traditional approach primarily revolves around the Seq2 Seq architecture. However, traditional Seq2 Seq has two significant drawbacks. Firstly,  
Figure 3: Bert Model Structure Diagram. at the encoder end, all information is compressed into a single vector, which includes all the information from the encoder end, leading to information loss. Additionally, the decoder end cannot use attention to focus on important information. Secondly, it cannot perform parallel computation, fundamentally similar to the reason why recurrent neural networks cannot be parallelized. Before formal encoding, the first step is tokenization. In the Bert model, tokenization consists of initial tokenization and subword tokenization. Initial tokenization converts the input text data into Unicode strings, while removing any illegal characters and extra spaces. It then distinguishes between Chinese and English. For Chinese text, each Chinese character is separated by a space, while English remains unchanged. Subsequently, unnecessary characters and spaces are removed, resulting in tokenization where each element is either a Chinese word, a regular English word, or a phrase. Subword tokenization uses a greedy longest-match-first algorithm on each basic token unit obtained from the initial tokenization. It further divides an English word or Chinese word into smaller units based on whether it is in the vocabulary. The token sequence is then marked with the CLS and SEP tokens to form the final text tokens. This paper adopts the approach of [PERSON] et al. (2018), using the WordPiece tokenizer for tokenization to obtain the corresponding tokens. Finally, after undergoing three embedding layers for tokenization, category, and positional aspects, the current text's word vectors are obtained. These are then fed into [PERSON]'s Transformer Encoder for encoding to obtain the text's sentence vector representation. Next, the Bert model is used to extract relevant sentence embeddings from low-level word tokens. The encoder in the Bert model maps the \(\left\{x_{1}^{t},x_{2}^{t}\ldots x_{n}^{t}\right\}\) of the sentence to a continuous representation sequence \(\left\{z_{1}^{t},z_{2}^{t},\ldots,z_{n}^{t}\right\}\), and then calculates the vector inner product at each word embedding position using convolution kernels with three window sizes. The convolution output of the \(k\)-th word under the \(m\)-th window is:  
\[p_{m,k}=Relu(w_{m}z_{kk+m-1}+b_{m}),m=\left\{1,2,3\right\} \tag{5}\]  
where \(w_{m}\) represents the convolution kernel matrix and \(b_{k}\) is the bias in the linear map. Due to the dimension difference of the convolution kernel, the word representation should be zero-padded to ensure that the dimension is not lost after convolution calculation. Max pooling is applied to all the words in the sentence:  
\[q=max(p_{m,1},\ldots,p_{m,n}) \tag{6}\]  
\(q\) represents the features obtained from a word sequence under the action of a convolutional kernel. This paper employs multiple convolutional kernels to consistently extract diverse features from the word sequence, which are subsequently passed to a fully connected layer.