# Joint Embedding of Semantic and Statistical Features for Effective Code Search
## 4 Experiment and Analysis
### Results Analysis  
#### 4.4.1 RQ 1: Effectiveness Evaluation of Code Search Approaches  
We verify the effectiveness of JessCS in terms of search model and search results. An automated evaluation method is used to evaluate the code search model on the validation set. During model training, we evaluate the model on 1000 validation sets every 5 iterations and record the TOP-1 precision and MRR values. UNIF [12] is a supervised extension model based on NCS [26]. This work draws on the ideas in UNIF to build and implement its network structure for comparative experiments, which is referred to as UNIF-based here. The two techniques, UNIF-based and JessCS, were trained for 1000 epochs each. According to the TOP-1 precision and MRR value recorded in the model training, the curve of the TOP-1 precision and MRR value of the two models with the number of model iterations were drawn. Figures 8 and 9 represent how the TOP-1 precision and MRR values change with the increase in iterations. According to the figures, the JessCS model converges when the iteration reaches about 500 th round and tends to stabilize after a slight decrease. The UNIF-based model converges when the iteration reaches the 700 th round, and then tends to be stable. In this study, the models with the best performance in the training process of these two different network structures are compared in Table 1. We measured the statistical indicators, i.e., \(p\)-value and standard error, for the values of TOP-1 precision and MRR in Figures 8 and 9. The significance level is set to 0.05 in the measurement. The \(p\)-values for Figures 8 and 9 are 0.45 and 0.39, which are much greater than the significance level. The corresponding standard errors are 0.0432 and 0.0448. These statistical indicators show that the TOP-1 precision and MRR values are statistically determined. Both of the two measurements can indicate the effectiveness of the studied approaches. As shown in Table 1, JessCS obtained the optimal model when iterated to 775 rounds, its TOP-1 precision was 0.708 and the MRR value was 0.791. The UNIF-based approach obtained the optimal model when iterating to 755 rounds, its TOP-1 precision was 0.578, and the MRR value was 0.682. The code search model proposed in this paper is 13% higher than UNIF-based approach in terms of the TOP-1 precision, and 0.109 higher than UNIF-based approach in terms of the MRR indicator. We collected 30 query sentences from the high-frequency query set of the StackOverflow (StackOverflow, [[https://stackoverflow.com/](https://stackoverflow.com/)]([https://stackoverflow.com/](https://stackoverflow.com/)) (accessed on 3 September 2020)) and the commonly used high-frequency queries in the existing research [10; 37; 38]. Table 2 presents the collected query descriptions. We use JessCS and UNIF-based approach to code search for each query statement. We use Frank to indicate the effectiveness of code search approach in real attempts. The \(NF\) result means the approach cannot search a correct code snippet. The summary of search results is shown in Table 3: \begin{table}
\begin{tabular}{c c c c} \hline  & **Iteration** & **TOP-1 Precision** & **MRR** \\ \hline JessCS & 775 & 0.708 & 0.791 \\ UNIF-based & 755 & 0.578 & 0.682 \\ \hline \end{tabular}
\end{table} 
Table 1: Performance comparison of UNIF-based and JessCS. Figure 8: The indicators of effectiveness for JessCS with the number of model iterations. Figure 9: The indicators of effectiveness for UNIF-based approach with the number of iterations. After evaluating the two code search models on 30 query statements, the query success rate and the average ranking reciprocal are calculated, as shown in Table 4: