# C2B: A Semantic Source Code Retrieval Model Using CodeT5 and Bi-LSTM
## 4 Proposed Architecture for C2B Hybrid Model
### The Main Building Blocks of the Proposed C2B Hybrid Model
#### 4.1.3 C2B Layer
-
* Cosine similarity attention: After calculating the cosine similarity between the query and code embeddings, the attention weight for each code token with respect to the query token can be obtained using \(softmax\) normalization. The attention weight, \(A_{ij}\), for the code token at position \(j\) given the query token at position \(i\) can be calculated using Equation (13). \[A_{ij}=softmax\big{(}Cosine\ Similarity\big{(}Q_{i},C_{j}\big{)}\big{)}\] (13) The \(softmax\) function ensures that the attention weights sum up to 1, creating a valid probability distribution over the code tokens for each query token. The cosine similarity attention formula measures the similarity between query and code embeddings based on the cosine similarity metric. It allows the model to assign higher attention weights to code tokens that are more similar to the query, enabling focused attention on relevant parts of the code during source code retrieval tasks. * Weighted code embeddings: Multiply each code embedding \(C_{j}\) by its corresponding attention weight \(A_{ij}\) to obtain the weighted code embeddings. This can be represented using Equation (14). \[Weighted\ Embedding_{j}=A_{ij}*C_{j}\] (14)
Each weighted embedding captures the importance or relevance of the corresponding code token in relation to the query token. * Aggregated code representation: Sum or average the weighted code embeddings to obtain an aggregated code representation. This step combines the information from different code tokens into a single representation, focusing on the relevant parts determined by the attention weights. The sum aggregation is computed using Equation (15a), and average aggregation is computed using Equation (15b). \[Agg_{Code}=Sum(Weighted\ Emb_{j})\ over\ all\ j \tag{15a}\] \[Agg_{Code}=Average(Weighted\ Emb_{j})\ over\ all\ j \tag{15b}\]  
The sum aggregation considers the total contribution of the weighted embeddings, while the average aggregation provides a normalized representation. The resulting \(Aggregated_{Code}\) representation captures the combined information of the code tokens that are most relevant to the query. It represents a condensed version of the code snippet, emphasizing the important aspects highlighted by the attention mechanism. By performing this aggregation step, the model effectively focuses its attention on the code tokens that have higher attention weights, indicating their relevance to the query. This aggregation enables the model to generate a comprehensive and contextually relevant code representation for source code retrieval tasks. In our C2B hybrid model, attention is calculated between the query and the code before the cosine embedding loss (CEL) and binary cross-entropy loss (BCE) loss. The query and code embeddings are generated using techniques like token embeddings and positional encodings, as discussed previously. The query and code embeddings are passed through attention mechanisms to calculate the attention weights. The attention mechanism allows the model to focus on the most relevant parts of the code given the query. The attention weights are used to weight the code embeddings, aggregating the relevant information from the code based on the query's attention distribution. The aggregated code representation is then used for prediction, and the CEL and BCE loss is calculated by comparing the predicted output with the ground truth labels. The detail of how cosine embedding loss (CEL) and binary cross-entropy loss (BCE) are computed is explained in Sections 4.2.1 and 4.2.2. After the aggregation step, the aggregated code representation can be used for prediction. For example, it can be fed into a classification or ranking model to predict the relevance or ranking of the code snippet. The BCE loss is then calculated by comparing the predicted output (relevance score or ranking) with the ground truth labels. The CEL and BCE loss measures the dissimilarity between the predicted and ground truth outputs and is used to update the model's parameters during training.