# Incorporating Code Structure and Quality in Deep Code Search
## 5 Evaluation
### Evaluation Method
#### 5.2.2 Evaluation Metrics  
We submit 45 questions to SQ-DeepCS and obtained the corresponding search results. In order to evaluate the results, five experienced developers were invited to determine whether the results can or cannot resolve the submitted questions. The decisions made by the developers are binary. When the developers hold different opinions on the results, they will discuss to reach an agreement.  
We use four evaluation indicators to measure the effectiveness of SQ-DeepCS: FRank, SuccessRate@k, PrecisionRate@k, and MRR. All of them are widely used in the research domain of information retrieval and code searching.  
FRank (also known as the best hit rank) is the rank of the first correct result in the query result list. The assumption behind FRank is that users often browse down from the first search result. The smaller the FRank of the search result, the less effort the users make to obtain correct results. We use FRank to evaluate the effectiveness of a single search and record the corresponding value of each query.  
SuccessRate@k (also known as the success percentage at _k_) represents the percentage of the queries for which more than one correct result could exist in the top \(k\) ranked results [31; 32; 33]:  
\[\textit{SuccessRate@k}=\frac{1}{|Q|}\sum_{q=1}^{Q}\delta(FRank_{q}\leq k) \tag{23}\]  
where \(Q\) is the number of all queries. \(\delta(cdot)\) is a function which returns 0 if the input is false and 1 otherwise. SuccessRate@k counts the number of queries with correct results in the first k results of all the queries and then divides it by the total number of queries. A good code search engine should help users find correct search results in shorter time so as to save their query cost. The higher the SuccessRate@k, the better the code search performances.  
Precision@k is a variant of accuracy which calculates the average value of the quality of query results. In our evaluations, it is calculated as:  
\[Precision@k=\frac{1}{|Q|}\sum_{q=1}^{Q}\frac{relavant_{q,k}}{k} \tag{24}\]  
where \(relavant_{q,k}\) represents the number of search results related to the query statement in the top k results of the \(q\)th query. Precision@k is important because developers often inspect multiple results of different usages to learn from [34]. A better code search algorithm should allow less noisy results so that users can obtain more relevant results. In our experiment, the higher the Precision@k, the better the code search performances. For FRank, Success@k, and PrecisionRate@k, we recorded their values when k is set to 1.5 and 10. For MRR, we only recorded the value when k equals 10.  
MRR is the average of the reciprocal ranks in the results of all queries. The reciprocal rank of a query is the reciprocal of the ranking of the first correct search result. MRR is calculated by the following formula:  
\[MRR=\frac{1}{|Q|}\sum_{q=1}^{Q}\frac{1}{FRank_{q}} \tag{25}\]  
The idea behind MRR is the reciprocal of the sorting results: the MRR score of the first result is 1, the score of the second result is 0.5, and the score of the nth match result is 1/n. In our experiment, the score of not finding (NF) the correct result in the top 10 returned results is recorded as 1/11. The higher the MRR value, the better the code search performance.