# Joint Embedding of Semantic and Statistical Features for Effective Code Search
## 3 Approach
### Joint Embedding of Semantic and Statistical Features
#### 3.2.1 Code Embedding Network  
Based on source code feature extraction and modeling, we propose a deep learning model that represents code fragments as continuous distributed vectors. They represent code fragments as a fixed-length code vector, which can be used to predict the semantic properties of code fragments. Deep learning models can help mine the relationship between query descriptions and code. Compared with traditional text retrieval methods, deep learning-based code search using deep neural networks to learn code and natural language representations can obtain more effective results [12; 25; 26]. Through this network model, the deep statistical features of the code can be extracted and applied to the downstream code search task. The code embedding network is shown in Figure 3. According to the triplet of code \(<\)method name, API invocation, token collection\(>\) obtained by modeling, we generate embeddings for them, respectively, and integrate the embedding vectors of three different features into a code representation vector \(E_{c}\). Method name embedding. We suppose the method name containing T words, which are represented as \(M=x_{1},x_{2},x_{3},\ldots,x_{T}\). The specific scheme is as follows:  
* Embedding layer: The role of this layer is to convert each word \(x_{i}\) into a corresponding vector representation \(e_{i}\). A word vector matrix \(W_{M}\in\mathbb{R}^{d\times|V|}\) is defined, where \(V\) is a fixed-size vocabulary, \(d\) is the dimension of the word vector, and \(W_{M}\) is the parameter matrix obtained by training. Each entry of the word vector matrix (a word), can be converted into a representation of its word vector: \(e_{i}=W_{M}v^{i}\), where \(v^{i}\) is a one-hot vector with dimension \(|V|\). Then, the method named \(M\) is transformed into a matrix \(emb_{M}=e_{1},e_{2},\ldots,e_{T}\) as the input of the next layer of the model. * Attention layer: This work uses the attention mechanism to assign normalized weights to each word according to its weight to generate sentence vectors. For the output generated by the previous layer \(emb_{M}=e_{1},e_{2},\ldots,e_{T}\in\mathbb{R}^{d\times T}\), we use the following formula to calculate the normalized weight \(\alpha_{i}\) of each word vector \(e_{i}\): \[\alpha_{i}=\frac{exp(a_{m}\cdot e_{i}^{T})}{\sum_{i=1}^{n}exp(a_{m}\cdot e_{i} ^{T})},\] (1) where \(a_{m}\) is a trainable d-dimensional vector. After calculating the normalized weights, we weighted and summed each individual word vector to obtain the vector representation of the method name \(E_{M}\): \[E_{M}=\sum_{i=1}^{T}\alpha_{i}\cdot e_{i}.\] (2) API invocation embedding. The API invoked in one method are sequential nature. In this paper, the network structure of Bi-LSTM + attention is used to generate embeddings for API invocation sequences, which not only considers the weight relationship between different words, but also retains the position information of the sequence. The specific network structure is shown in Figure 4.
* Embedding layer: Like the embedding layer at the method name embedding, this layer converts the sequence of API invokes into a matrix of real numbers and passes it to the next layer of the model. * Bi-LSTM layer: The word vector matrix input at the upper layer is converted into a hidden state matrix output. The LSTM structure of this layer includes an input gate, forget gate, cell state, and output gate: Input gate. Used to decide how much new information to add: \(i_{t}=\sigma(W_{x_{i}}x_{t}+W_{h_{t}}h_{t-1}+W_{c_{i}}c_{t-1}+b_{i})\), where \(x_{t}\) represents the current input, \(h_{t-1}\) represents the previous hidden state, and \(c_{t-1}\) represents the previous cell state. Forgotten door.Used to decide how much old information to discard: \(f_{t}=\sigma(W_{x_{f}}x_{t}+W_{h_{t}}h_{t-1}+W_{c_{f}}c_{t-1}+b_{f})\). Cell state.