# Joint Embedding of Semantic and Statistical Features for Effective Code Search
## 3 Approach
### Code Search Engine  
We combine the code embedding network, adjusted and optimized it, and proposed a new code search model JessCS.The specific mode structure is shown in Figure 6.  
JessCS divides the model into three sub-modules, which generate embeddings for code and natural language descriptions separately, and then use a cosine similarity function to measure the similarity between these two embedding vectors. After generating the embeddings for both the code and the natural language description, a similarity measure function is needed to measure the similarity between the two embedding vectors. This study uses the cosine function to measure the similarity between the two embedding vectors:  
\[cos(c,d)=\frac{c^{T}d}{\|c\|\|d\|}, \tag{7}\]  
where \(c\) and \(d\) are the code vector and query vector, respectively. The higher the similarity, the more relevant the code is to the description.  
Embedding both codes and descriptions into a unified vector space requires training a code search model. Figure 6 shows a schematic diagram of model training.  
When training the model, we construct each training instance as a triplet \(\langle C,D+,D-\rangle\). For each code fragment \(C\), there is a positive description \(D+\) (the correct description of \(C\)) and a negative description \(D-\) (the wrong description of \(C\)) that is randomly selected from the pool of all \(D+\). When trained on the triplet \(\langle C,D+,D-\rangle\), the code search model predicts the cosine similarity between \(\langle C,D+\rangle\) and \(\langle C,D-\rangle\) and minimizes the following loss function:  
\[\mathcal{L}(\theta)=\sum_{\langle C,D+,D-\rangle\in P}max(0,\epsilon-cos(c,d+ )+cos(c,d-)), \tag{8}\]  
where \(\theta\) represents the model parameter and \(P\) represents the training dataset. \(c\), \(d+\), and \(d-\) are the embedding vectors of \(C\), \(D+\), and \(D-\). The \(\epsilon\) represents the significance level in the validation experiment. It is set to 0.05 in this work. This loss function drives up the cosine similarity between the code snippet and its correct description.  
We use a large-scale corpus to train the model according to the above method, and continuously adjust and optimize the model parameters during the experiment to ensure the model achieves the best effect. Our models are built on Keras (Keras, [[http://keras.io/](http://keras.io/)]([http://keras.io/](http://keras.io/)) (accessed on 2 August 2020), an open-source deep learning framework. The hyperparameters we used in training are as follows: the number of hidden units in each direction of all LSTMs is set to 200; the dimension of the word embedding is set to 100; the dimension of  
Figure 6: The framework of the code search engine.  
the output vectors of all attention layers is set to 100; the number of hidden units of feature fusion MLP is set to 100. The JessCS is trained with the mini-batch-Adam algorithm [2; 27], the batch size (i.e., the number of instances per batch) is set to 128. We limit the vocabulary size to 10,000 words, which are the most used words in the training dataset. Through this training process, JessCS can embed code and natural language descriptions into a unified vector space.  
JessCS focuses on comprehensive extraction of deep statistical features from source code and performing semantic analysis and modeling on source code. DeepCS [4] uses CODEnn model to represent code snippets with some directly available descriptions (e.g., high-quality comments or keywords). Due to the lack of deep investigation on code representation, DeepCS is outperformed by UNIF-based approach in recent study [12]. So we select the UNIF-based approach to make the comparison in our experiments.