# A Context-Aware Neural Embedding for Function-Level Vulnerability Detection
## 4 Related Work  
### Embedding Methods for Source Code  
The bimodal property of programming languages: understood by computers and readable by humans has motivated researchers to apply NLP techniques, including neural language models to software code analysis [16; 17; 48]. Different from conventional ML techniques, which primarily rely on labor-intensive feature engineering, neural network models are capable of learning latent patterns which can be more generalizable to the task of interest. However, there is a \"domain gap\" between the neural network, which was originally designed to cope with the numeric data and software code, which consists of textual tokens being \"discrete\" in nature. To bridge the gap, various code encoding methods were used. One-hot encoding was adopted by many studies which applied neural techniques to code analysis. [PERSON] et al. [19] used one-hot encoding for representing source code tokens. One-hot encoding was also used at the character level [20; 21]. Unfortunately, one-hot encoded vectors fail to capture the similarities/differences of the items they represent. Hence, different methods were proposed to generate code embeddings as distributed representations. In the field of NLP, the distributed representations of words (e.g., Word2 Vec) capture the meanings of words distributed in the components of vectors [16]. With Word2 Vec model, words can be encoded, using meaningful vector representations to better facilitate the learning tasks. For this reason, many existing studies that applied neural models for code analysis used Word2 Vec model for learning code embeddings (see, for example, [14; 25; 26; 27; 49; 50; 51]). Embedding techniques, such as Word2 Vec, failed to generate different representations for the same word used in different contexts. To better optimize the learning of contextual representation of words, methods such as Context2 Vec [29] and ELMo [15] were proposed. To transfer source code to vector representations, [16; 52; 53] applied deep learning techniques to generate code embeddings based on ASTs. Our work differs with the aforementioned studies in two main points: firstly, our trained framework takes directly source code as inputs, which does not require any code analysis tools for further processing and outputs the learned function-level embeddings. Secondly, our framework applies ELMo, which is a contextualized model, for generating code representations based on contexts. Additionally, the Bi-LSTM structure also helps to capture the long-range dependencies of sequences. Due to vulnerability detection requiring the understanding of code contexts to better track the data flows and/or control flows, our proposed Bi-LSTM framework incorporating ELMo can better identify vulnerable patterns related to contexts. The aforementioned embedding methods originate from the field of NLP. There is a line of studies which extract the code embeddings from the syntactic tree representations and intermediate representations (IRs) of software code. [PERSON] et al. [16] proposed Code2 Vec, which decomposes a source code function to a collection of paths in its AST, allowing a neural network to learn the representation of each path and use the attention mechanism to select the relevant paths and aggregate a set of them to form a fixed-length vector. Later, the authors proposed code2 seq [54], which is an improved method for representing source code with the syntactic structure of programming languages. However, both Code2 Vec and Code2 Seq are programming language dependent. [PERSON] et al. [44] proposed inst2 vec, which produces code representations based on IRs, which are based on the data flow and control flow of a program, making the generated representations programming language independent. To facilitate reverse engineering, Asm2 Vec was proposed to generate robust representations for assembly code, which can prevent changes brought by obfuscation and optimizations [55]. Similarly, [PERSON] et al.