# C2B: A Semantic Source Code Retrieval Model Using CodeT5 and Bi-LSTM
## 5 Experimental Results
### Results and Discussions
#### 5.4.1 Experiment 1  
We divide the experiment into two parts. First, we present and compare the results of two of our proposed training strategies--CE and BCE losses. Second, we compare the results of our best model with the existing works against the clone detection metrics defined in the previous subsection. * Experiment 1.1: Table 3 depicts the results of our proposed C2B model on two training strategies along with the original CodeT5 results on the clone detection task. The first row in the table represents the results of the original CodeT5 architecture (with concatenated code tokens as described in Section 4.1.3) on the clone detection task. The second row represents the original architecture directly used without any fine-tuning on single code input. The third and the fourth row represent the modified CodeT5 architecture fine-tuned on CE and BCE losses, respectively. As we can see from Table 3, the original CodeT5 performs the best on all metrics compared with the original Bi-LSTM and our proposed C2B model on BCE and CE losses. One reason why our fine-tuned C2B model lags behind the original CodeT5 architecture can be attributed to the fact that the CodeT5 authors use a maximum code token length of 400. In contrast, we truncate the maximum code token length to 128 to stay within our computational limits, thus losing a lot of useful information. Note that the original CodeT5 results were taken from the Github [[https://github.com/salesforce/CodeT5/issues/55#issuecomment-1178517051](https://github.com/salesforce/CodeT5/issues/55#issuecomment-1178517051)]([https://github.com/salesforce/CodeT5/issues/55#issuecomment-1178517051](https://github.com/salesforce/CodeT5/issues/55#issuecomment-1178517051)) (accessed on 15 April 2024) repository and not from their work [18], as they claimed that they reported an incorrect result in the paper. We can see from Table 3 that in contrast to the CodeT5 model, the original Bi-LSTM model's performance seems to be lower in terms of precision, recall, and F1 score. This indicates that it may struggle to capture relevant information effectively. It can be clearly observed in the table that the proposed C2B model using BCE loss achieves a higher F1-score (0.9023) than using CE loss (0.7485). Thus, we can conclude that the proposed C2B model, by minimizing BCE loss, tends to provide better code embeddings than CE loss. To verify this claim, we perform Experiment 2, in which we visualize the code embeddings and generate meaningful interpretations. * Experiment 1.2: In this experiment, we compare the performance of our proposed C2B model with some classical code search approaches from the literature. The results are presented in Table 4. All the results are generated on the CodeSearchNet testing dataset. The results of the existing works are arranged in ascending order of F1-scores. We can observe that the classical code search approaches such as NBOW [91] and CNN [92] have high precision but low recall values and thus a lower F1-score. Works
such as BiRNN [93] and SelfAtt [94] are classical deep learning architectures that obtain better precision and recall scores than NBoW and CNN. Recent deep learning architectures such as RoBERTa [56], CodeBERT [55], GraphCodeBERT [54], CodeT5 [18], and FA-AST [53] compete to attain top results on the task. Three architectures--FA-AST, GraphCodeBERT, and CodeT5--currently achieve state-of-the-art F1-scores (0.95) on the clone detection task. Our best model, which is the modified CodeT5 architecture fine-tuned on BCE loss, obtains an F1-score of 0.902. The main reason why our best model drops in the F1-score is because we use a maximum code token length of 128 as opposed to all the other architectures which use a maximum code sequence length greater than 400. Our focus was never to achieve top results in clone detection but to assess whether the CodeT5 embeddings are capable of capturing essential semantic and syntactic code information. Our secondary aim was to identify whether the code embeddings could be useful for clone detection and retrieval.