## 5 Expertise Retrieval
### Measuring Similarity Between Mobile Apps' Descriptions
#### 5.1.2 App Description Similarity Setup
In addition, when the vectors \(x\) and \(y\) point in opposite directions and the angle between the vectors is equal to \(180^{\circ}\), this gives a cosine similarity \(cos\) (\(\theta\)) = \(-1\) which means the vectors are opposite [87]. Figure 4 [87] shows the three main cases of cosine similarity. Since we do not have a benchmark dataset where pairs of apps' descriptions are annotated as either similar or not similar or related and not related, another approach has been adopted to evaluate the reliability of the generated similarity scores which is human judgment. Human judgments were used to measure the similarity between a query description and 15 apps descriptions where the descriptions are taken from the category (e.g., Games) mentioned in the query. The pair to be measured is a query description (e.g., \(q_{2}\) in Table 2) and an app description from the collection of 15 descriptions selected from the Games category. The fifteen descriptions are selected by choosing from the six query descriptions: five considerably similar descriptions to the query description, five moderately similar descriptions and five dissimilar descriptions. Three human raters rated each of the 15 pairs for each query with a score from 0 to 10. They were asked to judge how similar the pairs are in terms  
Figure 4: Cosine similarity cases. of the specific domain and features as well. If the two applications have similar domains (e.g., both are applications for diabetes monitoring) this would indicate a considerably high similarity. Additionally, they were asked to pay attention to the features; if they have similar features this raises the similarity score, even if the compared applications do not have the exact same domain. A total of 90 pairs were rated by three raters, and the average of the rates was calculated to end up with a final human judgment (H.J.) score for each pair. Furthermore, to find the most reliable model that can detect how similar apps' descriptions are to each other, the correlation coefficient is measured between the averaged human judgment scores and the cosine similarity score generated by the pre-trained SBERT and Doc2 Cec models. The correlation coefficient formula is shown below, in which \(cov(x,y)\) is the covariance between two variables \(x\) and \(y\), and \(\sigma_{x}\)\(\sigma_{y}\) is the standard deviation for the variables \(x\) and \(y\). \[\text{Correlation coefficient}=\frac{cov(x,y)}{\sigma_{x}\ \sigma_{y}}\]  
The correlation coefficients between the averaged human judgments and SBERT similarity scores are plotted in Figure 5 (left column) for all queries, and the correlation coefficients between the averaged human judgments and Doc2 Vec similarity scores are plotted in Figure 5 (right column) for all queries. The plots show that there is a strong positive correlation between the averaged human judgments and the SBERT similarity scores, which is evidently stronger than the relation between the averaged human judgments and Doc2 Vec similarity scores. Figure 5: _Cont._  
The difference between the correlation coefficient achieved by SBERT and Doc2 Vec with H.J. is visualized in Figure 6. In addition, the detailed numerical results are illustrated in Tables 3-8 for \(q_{1},\,q_{2},\,q_{3},\,q_{4},\,q_{5}\,\) and \(q_{6}\), respectively. The tables show the cosine similarity between the 15 selected apps' descriptions and the corresponding query description for each of the six queries when applying SBERT and Doc2 Vec. In addition, they show the average of human judgments (H.J.) for the 15 pairs and the correlation coefficient (C.C.) between H.J. and SBERT and H.J. and Doc2 Vec scores. All 90 apps' descriptions used in this experiment are available in [88]. The results revealed that SBERT is superior to Doc2 Vec since it is strongly correlated to human judgments. SBERT has proved its effectiveness in measuring similarity and relatedness between mobile apps' descriptions.