# C2B: A Semantic Source Code Retrieval Model Using CodeT5 and Bi-LSTM
## 4 Proposed Architecture for C2B Hybrid Model
### Training Procedure
#### 4.2.1 Cosine Embedding Loss (CEL)  
In this training strategy, we aim to bring the embeddings of the true clone pairs closer to the common embedding space and simultaneously push the embeddings of nonclone pairs farther from each other. We use a cosine similarity metric to determine the level of similarity between each code pair. Using the cosine similarity, we define a cosine embedding loss and plan to minimize the loss by updating the model parameters. In this way, the model learns to project the code into an embedding vector rich in code syntax and semantics.  
* Cosine similarity (\(C_{Sim}\)): It represents the level of likeness between two vectors. In our case, we project two codes into embedding vectors using the architecture to compute the cosine similarity. If the two vectors point in the same direction, they have a high level of similarity, and vice versa. The \(C_{Sim}\) is essentially the dot product of \(V_{1}\) and \(V_{2}\) normalized by the product of their magnitude. The \(C_{Sim}\) outputs a floating point value in the range [0, 1], where the value 0 (1, resp.) indicates the highest level of dissimilarity (similarity, resp.) between vectors. Given two vectors A and B, the cosine similarity is defined by Equation (16). \[Cosine\ Similarity(V_{1},V_{2})=\cos(\theta)=\frac{V_{1}\cdot V_{2}}{\|V_{1}\| \|V_{2}\|}\] (16)
* Cosine embedding (CE) loss: The CE loss is derived from the cosine similarity. This loss is used to measure whether the two vectors are similar or dissimilar. If the two vectors of true (false, resp.) clone pairs are predicted dissimilar (similar, resp.), then the loss penalizes the model parameters. In other words, if the prediction and the actual labels do not match, the loss function takes a high value. The CE loss for each sample is defined in Equation (17). \[l_{CE}(z_{1},z_{2},TL)=\begin{cases}1-\cos(z_{1},z_{2}),&if\ TL=1\\ max(0,\cos(z_{1},z_{2},\gamma)),&otherwise\end{cases}\] (17) where \((z_{1},z_{2})\) are two embedding vectors, \(TL\) is the true label, and \(\gamma\) is the margin, which is a tunable hyperparameter. We use the default value of the margin \((\gamma=0)\). The CE loss encourages the cosine angle between the embedding vectors to be small if both vectors are similar [86].
* Fine-tuning C2B hybrid model parameters on CE loss. Figure 4 depicts how the architecture is fine-tuned by minimizing the CE loss. We instantiate a single pretrained  
Figure 3: Training of C2B hybrid model.  
CodeT5 architecture and input code-1 and code-2 tokens to extract their embeddings individually. Thus, there is parameter sharing (single CodeT5 instance) as illustrated in the figure. Once, the code embeddings are generated, the CE loss is computed and the model parameters are updated using gradient descent. The gradient descent [87] is an optimization algorithm to adjust the model parameters in order to minimize the loss function.
* Inference on clone detection: During clone detection inference, we input the code and query tokens to the fine-tuned architecture. Specifically, we input code and query tokens individually to the model to extract their embeddings. The cosine similarity between the embeddings is computed, which determines the level of similarity between the code and query. Finally, we compare the similarity score within a certain threshold. If the similarity lies above the threshold, the code is claimed to be a true clone and vice versa.
* Inference on C2B hybrid model code retrieval: Once the model is fine-tuned, we precompute and store the embeddings corresponding to all the codes and query within the dataset. During code retrieval inference, the input is a single query. We use the architecture only once to extract the query embeddings. We then compute the cosine similarity between the query embeddings and all the precomputed embeddings. The codes in the dataset are ranked according to their similarity with the query. Finally, we choose and retrieve the top-k codes as the clone pairs.