# C2B: A Semantic Source Code Retrieval Model Using CodeT5 and Bi-LSTM
## 4 Proposed Architecture for C2B Hybrid Model
### The Main Building Blocks of the Proposed C2B Hybrid Model
#### 4.1.2 Word Embeddings
In CodeT5, the transformer-based architecture plays a crucial role in capturing the contextual relationships and dependencies between the tokens. The model consists of multiple layers of self-attention and feedforward neural networks. After generating the query and code embeddings, CodeT5 can leverage these embeddings for various retrieval tasks, such as similarity matching or ranking of relevant code snippets based on the query. The embeddings capture both the semantic meaning of the tokens through the token embeddings and the positional information through the positional encodings. This enables the model to effectively understand and retrieve relevant source code in the context of the given query. * Bi-LSTM word embeddings: The generated word embedding as shown in Figure 2 are fed into the Bi-LSTM layer to capture the contextual information of query and code as follows: * Query embedding: Let us denote the query token at position \(i\) as \(Q_{i}\). The embedding for the query token can be obtained using an embedding layer, which maps the token to a continuous vector representation. We can represent this using Equation (7). \[Q_{i_{embedding}}=Embedding(Q_{i})\] (7) The embedded query tokens are fed into the Bi-LSTM layer. The Bi-LSTM processes the tokens in both the forward and backward directions, capturing the contextual information. The forward hidden state at position \(i\) is denoted as \(H_{i_{forward}}\), and the backward hidden state is denoted as \(H_{i_{backward}}\), as shown in Equations (8a) and (8b). \[H_{i_{forward}}=LSTM\Big{(}Q_{i_{forward}},H_{(i-1)_{forward}} \Big{)}\] (8a) \[H_{i_{backward}}=LSTM\Big{(}Q_{i_{forward}},H_{(i+1)_{backward}} \Big{)}\] * Code token embedding: Similar to the query token embedding, the code token at position \(i\), \(C_{i}\), can be embedded using the same embedding layer. The embedding for the code token is represented using Equation (9). \[P_{Q_{i,k}}=sin\left(\frac{Q_{i}}{10,000^{2^{k/d}\_model}}\right)for\;\;even\;k\]0 (9)The embedded code tokens are fed into the Bi-LSTM layer, which processes them in both the forward and backward directions, generating forward hidden states, \(H_{i_{forward}}\), and backward hidden states, \(H_{i_{backward}}\). \[P_{Q_{i,k}}=sin\left(\frac{Q_{i}}{10,000^{2^{k/d}\_model}}\right)for\;\;even\;k\]1 \[P_{Q_{i,k}}=sin\left(\frac{Q_{i}}{10,000^{2^{k/d}\_model}}\right)for\;\;even\;k\]2  
In Equations (10a) and (10b), \(Embedding()\) represents the embedding layer that maps tokens to continuous vector representations. \(LSTM()\) denotes the long short-term memory layer, which processes the inputs (query tokens or code tokens) and the hidden states from previous time steps to generate the current hidden states. The Bi-LSTM processes the tokens sequentially, incorporating both the previous and future contextual information. The forward hidden state, \(H_{i_{forward}}\), is updated using the token embedding and the previous forward hidden state. Similarly, the backward hidden state, \(H_{i_{backward}}\), is updated using the token embedding and the subsequent backward hidden state. By iteratively applying these formulas to all the tokens in the query or code, you can obtain the query and code embeddings using Bi-LSTM. These embeddings capture the contextual information and semantic meaning of the inputs, enabling effective source code retrieval based on query matching and similarity.