# C2B: A Semantic Source Code Retrieval Model Using CodeT5 and Bi-LSTM
## 4 Proposed Architecture for C2B Hybrid Model
### The Main Building Blocks of the Proposed C2B Hybrid Model
#### 4.1.2 Word Embeddings  
There are two main parts of this block: CodeT5 word embeddings and Bi-LSTM word embeddings. * CodeT5 word embeddings: Our proposed C2B hybrid model is illustrated in Figure 2. Instead of taking two codes as input to the model, we generate embedding for each code individually and query (or code comments). For the code search task, we compute raw embeddings for both code and query separately and then experiment with two different training strategies to learn these embeddings. Once the embeddings are trained, we compute and store them for every code in the dataset. Our idea is to use the code and query embeddings for comparison during code retrieval rather than using the whole architecture. This proposed approach is based on the assumption that the embeddings capture syntactic and semantic information present in the program and query. We conduct experiments to assess the efficacy of these embeddings. If the assumption holds true, then the code retrieval task reduces to comparing these embeddings, which is faster and more efficient than using architecture for pairwise comparison. CodeT5, an extension of the T5 (text-to-text transfer transformer) model, can generate query and code embeddings for source code retrieval tasks. Here is a detailed explanation of how CodeT5 generates query and code embeddings for our C2B hybrid model, along with the formulas involved:
* Input encoding: The first step is to tokenize the query and code into subword units using techniques like WordPieces or byte-pair encodings (BPEs). Let us denote the tokenized query as \(Q=[Q_{1},Q_{2},...,Q_{n}]\) and the tokenized code as \(C=[C_{1},C_{2},...,C_{m}]\). * Embedding layer: The tokenized query and code are passed through an embedding layer, which maps each token to a continuous vector representation. The embeddings for the query tokens and code tokens are denoted as \(E_{Q}=[E_{Q1},E_{Q2},...,E_{Qn}]\) and \(E_{C}=[E_{C1},E_{C2},...,E_{Cm}]\), respectively. These embeddings capture the semantic meaning of the tokens. * Positional encoding: To incorporate positional information, positional encodings are added to the query and code embeddings. Positional encodings help the model understand the relative and absolute positions of tokens within the input sequences. The positional encodings for the query and code are denoted as \(P_{Q}=\left[P_{Q1},P_{Q2},...,P_{Qn}\right]\) and \(P_{C}=\left[P_{C1},P_{C2},...,P_{Cm}\right]\). Each positional encoding is a vector that represents the position of the token within the sequence. The positional encoding vectors can be computed using sine and cosine functions as shown in Equations (1)-(4). \[P_{Q_{i,k}}=sin\left(\frac{Q_{i}}{10,000^{2^{k/d}\_model}}\right)for\;\;even\;k\] (1) \[P_{Q_{i,k}}=cos\left(\frac{Q_{i}}{10,000^{2^{k/d}\_model}}\right) for\;\;odd\;k\] (2) \[P_{C_{i,k}}=sin\left(\frac{Q_{i}}{10,000^{2^{k/d}\_model}}\right) for\;\;even\;k\] (3) \[P_{C_{i,k}}=cos\left(\frac{Q_{i}}{10,000^{2^{k/d}\_model}}\right) for\;\;odd\;k\] (4)
Here, \(Q_{i}\) and \(C_{i}\) represent the position of the token within the query and code sequences, respectively. \(d_{model}\) refers to the dimensionality of the embedding space, and \(k\) represents the dimension index of the positional encoding vector. * Query embedding: The query embedding is obtained by summing the token embeddings and positional encodings for the query tokens. This can be represented using Equation (5). \[Query_{Embedding}=sum\big{(}E_{Q}+P_{Q}\big{)}\] (5) The sum operation combines the token embeddings and positional encodings for each token in the query, resulting in an overall representation of the query. * Code embedding: Similar to the query embedding, the code embedding is obtained by summing the token embeddings and positional encodings for the code tokens. This can be represented using Equation (6). \[Code_{Embedding}=sum(E_{C}+P_{C})\] (6) The sum operation aggregates the token embeddings and positional encodings for each token in the code, generating an overall representation of the code.