# C2B: A Semantic Source Code Retrieval Model Using CodeT5 and Bi-LSTM
## 4 Proposed Architecture for C2B Hybrid Model
### Training Procedure
#### 4.2.2 Binary Cross-Entropy Loss  
In this strategy, we train the architecture to perform clone detection and automatically learn the parameters to generate the code embeddings during the process, as shown in Figure 5. This strategy differs from the previous one conceptually and also by the way of computing loss on the embedding vectors. Let us first define and state the characteristics of the loss function. Figure 4: Embedding learning code embeddings by minimizing cosine embedding loss. * Binary cross-entropy (BCE) loss: It is a common loss function used in binary classification problems. In our case, given a code pair, we aim to predict whether they are true clone pairs or not. The BCE loss penalizes the model by returning a high value for every wrong prediction. It can be represented by Equation (18). \[l_{BCE}(y,\widehat{y})=\frac{1}{N}[y_{i}\cdot\log(\widehat{y}_{i}+(1-y_{i})) \cdot(1-\widehat{y}_{i})]\] (18) where \(N\) is the total number of samples, \(y\) is the true label (\"1\" if true clone pair, otherwise \"0\"), and \(\widehat{y}\) is the predicted label. We train the model to minimize the loss and automatically learn the parameters to produce code embeddings. * Fine-tuning C2B model parameters on BCE loss: Figure 5 illustrates how the architecture is fine-tuned by minimizing the BCE loss. The whole process is similar to fine-tuning the architecture on CE loss till the point the embeddings are extracted individually. After that, the code embeddings are concatenated horizontally and given to a feedforward neural network (classifier) for classification. The output of the classifier is a probability indicating the level of similarity between the code pair. * Bi-LSTM network: Bi-LSTM is used in our model architecture. In the original CodeT5 framework, a two-layered feedforward neural network was employed, which accepted a single 768-dimensional embedding as input. In our adapted architecture, we use five-layer structure with nonlinear functions following each layer, similar to the original CodeT5 model. However, the key difference lies in the input dimensions, where we concatenate code and query embeddings, each comprising a 768-dimensional vector. This results in a 1536-dimensional input vector for our Bi-LSTM model. Our five-layer classifier architecture, featuring Bi-LSTM, is presented in Table 1. The Bi-LSTM model operates differently from the original feedforward neural network, as it processes sequential data by considering the sequential dependencies within the concatenated code embeddings. The sigmoid function continues to play a role in transforming raw scores (logits) into probabilities within the range [0, 1]. Notably, this classifier takes concatenated code embeddings as input and computes a weighted average over these embeddings, followed by the application of nonlinear activation functions (_Tanh_ or _Sigmoid_). Finally, the softmax pooling layer takes the probability values from the sigmoid layer and normalizes them across the entire dataset to ensure that the final  
Figure 5: Learning code embeddings by minimizing binary cross-entropy loss. output is a probability distribution, which helps in ranking and retrieving the most relevant code snippets. In total, the classifier consists of 1,181,954 trainable parameters. It is important to emphasize that the entire architecture including the classifier network is trained in an end-to-end fashion to optimize performance. The detailed network diagram is shown in Figure 6.  
* Inference on code search: We use the CodeT5 architecture to generate embeddings corresponding to both the codes individually. Further, our Bi-LSTM network predicts a probability indicating the level of similarity between the codes and query. If the probability is greater than a threshold (\(p\geq 0.5\)), then we classify the code as true for the given query. * Inference on code retrieval: We precompute and store the embeddings using the fine-tuned architecture on the codes present in the entire dataset. During inference, we fetch the query embeddings using the architecture.