# C2B: A Semantic Source Code Retrieval Model Using CodeT5 and Bi-LSTM
## 2 Literature Review
### Deep Learning Approaches
[53] used graph neural networks (GNNs) to search similar code snippets. They proposed FA-AST (flow-augmented AST), a technique that leverages control and data flow graphs to enhance the ASTs. The authors employed GNNs on FA-AST to measure the similarity between the code pairs. [PERSON] et al. [54] developed GraphCodeBERT to address the limitation of CodeBERT [55], which ignores the code's inherent structure. They used the program's data flow graph to encode semantic information about dependencies between variables. GraphCodeBERT is pretrained on three tasks: masked language modeling, predicting dependency edges in the code structure, and aligning representations between source code and code structure. The model improves CodeBERT results and achieves state-of-the-art performance on four downstream tasks. [PERSON] et al. [18] developed CodeT5, an encoder-decoder-based transformer model that uses user-defined identifiers to capture semantic properties in code. They extended a Seq2 Seq-based model for code understanding and generation applications and proposed a novel identifier-aware pretraining objective. CodeT5 outperformed prior architectures on fourteen code-related subtasks. [PERSON] et al. [56] proposed ROBERTA, a transformer-based language model designed to understand context and relationships in natural language and code. It utilizes a bidirectional self-attention [57] mechanism and large-scale pretraining on diverse data. ROBERTA has demonstrated strong performance in various NLP and source code-related tasks. [PERSON] et al. used a tree-based RNN model to represent code sequential content and integrate context vectors for summaries. Context vectors are high-dimensional numerical representations that capture the essential information and context of input data, often used in machine learning models to summarize or encode features. They also used reinforcement learning to address exposure bias, a common challenge in training models on imbalanced datasets. [PERSON] et al. [58] and [PERSON] et al. [59] shared a similar objective, which involves combining the contexts of the symbolic bag-of-words tree and the token sequence of the code to produce summaries. [PERSON] et al. [16] and [PERSON] et al. [60] used neural bag-ofwords (NBoW), a classic approach in source code retrieval that represents code snippets as fixed-size vectors by summing the embeddings of individual tokens. It is known for its simplicity and efficiency but struggles with capturing semantic relationships and context in code due to its bag-of-words nature. [PERSON] et al. [61] utilized an API sequence as an additional input in their model, while [PERSON] et al. [62] suggested using a convolutional neural network to extract vector representations of AST nodes and learn an adaptable weight vector across various code representations. Following a methodology similar to that presented in [52], [PERSON] et al. [63] adopted a similar strategy by incorporating retrieved samples to enhance their task. However, they went beyond utilizing the retrieved code snippet alone, also exploiting its abstract syntax tree (AST) and the associated comment, termed as the exemplar. Concurrently, there has been a growing interest in the utilization of graph neural networks (GNNs). [PERSON] et al. [64] used lexical information and AST for graph representation of code and then used GNN to encode this graph. To introduce sequential information into the GNN, they initialized a graph node with the corresponding output of the RNN. Expanding on this notion, [PERSON] et al. [65] amalgamated diverse representations of source code, encompassing AST, control flow graph (CFG), and program dependency graph (PDG), into a unified code property graph, leveraging this joint representation for generating summaries. Similarly, building upon their earlier research [58], [PERSON] et al. [66] employed a GNN for the AST as a distinct input alongside the token sequence. [PERSON] et al.