# Incorporating Code Structure and Quality in Deep Code Search
## 5 Evaluation
### Results  
Table 1 shows the evaluation queries and the corresponding FRank(DCS: DeepCS; AD: A-DeepCS; PD: P-DeepCS; PAD: PA-DeepCS; SQD: SQ-DeepCS; NCS and At-CodeSM are not given in the table due to space limitations). In Table 1, 'NF' represents Not Found, which means there is no relevant result in the top K results in this query (here we consider K as 10). A FRank value of 1 indicates that a user can obtain reusable code snippets in the first search result. The FRank value of 1 that appeared in DeepCS and SQD is 8 and 10. The number of 'NF' in DeepCS and SQD is 20 and 16. Therefore, SQD outperforms DeepCS from the perspective of FRank. The FRank value of 1 that appeared in AD, PD, and PAD is 6, 7, and 9. The number of 'NF' in AD, PD, and PAD is 22, 21, and 16. The results show that AD and PD perform even worse than DeepCS, while PAD outperforms DeepCS a bit but still worse than SQD. For NCS and At-CodeSM, the number of 'NF' is 20 and 17. In this regard, the effect of NCS is similar to that of DeepCS, slightly inferior to that of At-CodeSM, while SQD still maintains a good effect. The FRank value of 1 that appeared in NCS and At-CodeSM is 7 and 9. It means that when we only consider the top search results, SQD still performs well. Figure 6 shows the box-plot of FRank for the several approaches. The vertical axis represents the FRank values from 1 to 11, where we regard 'NF' as the FRank value of 11. The symbol '+' and the horizontal line in the box represent the mean and median of FRank. We can observe that the average FRank score of SQ-DeepCS is 6.09, which is lower than the average FRank score of DeepCS (6.94). This result shows that, from the perspective of the ranking of first useful search results, we have improved about 0.85 on the basis of DeepCS. The average FRank scores of AD, PD, and PAD are 6.96, 7.24, and 6.51. For NCS and At-CodeSM, the average FRank scores are 7.09 and 6.22. The results show that at the FRank level, the performance of SQD is ahead of other methods. Table 2 shows the overall accuracy of SQ-DeepCS and related approaches. The results show that the performance differences of A-DeepCS, P-DeepCS, and DeepCS are not significant. For example, S@10 of PD is 0.533, which is 2.3% lower than that of DeepCS. From these results, we could see that the introduction of program slice or attention mechanism may not improve the performance of DeepCS. Compared with DeepCS, S@k of PAD are improved by 4.4%, 2.2%, 8.8%, and P@k of PAD are improved by 4.4%, 1.8%, and 2.5%. For MRR, the improvement to DeepCS is 3.3%. The result shows that PA-DeepCS outperforms DeepCS a bit. Figure 6: The box-plot of FRank for SQ-DeepCS and related approaches. SQ-DeepCS outperforms PA-DeepCS in most indicators expected for P@10. The key difference of PA-DeepCS and SQ-DeepCS is that SQ-DeepCS adopts an attention layer to concentrate the vectors of method name, program slice, and tokens. The result implies that considering the quality of different parts of code snippets can significantly improve the search performance. By using attention mechanism when fusing method name, API sequence, and tokens of the code snippet, SQ-DeepCS gives higher weight to the high-quality parts, so as to better improve the search performance. For NCS and At-CodeSM, we can see that SQ-DeepCS once again takes the lead in the search performance evaluated by S@k, P@k, and MRR. Compared with NCS, MRR of SQ-DeepCS is improved by 12.3%. In addition, it is worth noting the difference between PA-DeepCS and At-CodeSM. PA-DeepCS outperforms At-CodeSM in most indicators. The key difference between them is the representation method of code snippets. The result