## 5 Expertise Retrieval
### Crowd Expertise Retrieval Algorithm Design  
According to the evaluation we have conducted to find an effective model that can be harnessed for information retrieval tasks, SBERT and specifically the pre-trained general-purpose model all-mpnet-base-v2 have proved its effectiveness. The results have shown that the pre-trained model is effective in measuring the similarity and relatedness between the mobile app textual descriptions. Therefore, to design the crowd retrieval algorithm, this model was utilized for measuring similarity between apps' descriptions. The retrieval algorithm was designed to retrieve a subset of the identified crowd of mobile apps users and developers that would most likely possess the application domain knowledge for a certain query mobile app. In fact, to limit the size of the retrieved crowd, only mobile app users and developers who have used or developed apps that are similar to the query mobile app submitted by the crowdsourcing requester are retrieved. For that, the cosine similarity score measured between app descriptions was set to a threshold = 0.30. This value was observed to be the best minimum score that would most likely assist in retrieving a crowd that have used or developed mobile apps similar to the query app. A discussion was held with two of the participating human raters, who have previously contributed to rating the similarity between descriptions, to decide on a suitable value for the threshold. The discussion yielded a consensus among the raters on the threshold value of 0.30. After the exchange of thoughts, it was noticed that mobile apps' descriptions with a score higher than 0.30 can be considered similar to the query description. The higher the score, the more likely the app would be similar to the query app. Scores between 0.45 and 0.30 would indicate a less similar app; however, between these scores it was observed that there is a possibility that a similar and related mobile app can be found. Eventually, the group discussion arrived at a conclusion that app descriptions with similarity scores below 0.30 are usually unrelated. \begin{table}
\begin{tabular}{c c c c c} \hline  & \multicolumn{4}{c}{\(q_{5}\)} \\ \cline{2-5}  & **App Name** & **SBERT** & **Doc2 Vec** & **H.J.** \\ \hline D13 & RevenueCat Notification Client & 0.08 & 0.39 & 0.17 \\ \hline D14 & Color Pro Picker & 0.10 & 0.33 & 0.67 \\ \hline D15 & CIDR Trainer & 0.11 & 0.37 & 0.17 \\ \hline C.C. & Correlation Coefficient with H.J. & 0.959 & 0.748 & - \\ \hline \end{tabular}
\end{table} 
Table 7: _Cont._ \begin{table}
\begin{tabular}{c c c c c c} \hline  & \multicolumn{4}{c}{\(q_{6}\)} \\ \cline{2-5}  & **App Name** & **Category** & **SBERT** & **Doc2 Vec** & **H.J.** \\ \hline D1 & 1 Big App & Utilities & 0.69 & 0.43 & 6.67 \\ \hline D2 & piZap: Design \& Edit Photos & Graphics \& Design & 0.61 & 0.43 & 7.67 \\ \hline D3 & CreativeApp & Personalization & 0.59 & 0.53 & 5.33 \\ \hline D4 & Photo Video Maker–PixPoz & Photo \& Video & 0.58 & 0.57 & 7.83 \\ \hline D5 & Magical Fraser & Photo \& Video & 0.58 & 0.38 & 7 \\ \hline D6 & Clip Studio Paint for iPhone & Graphics \& Design & 0.39 & 0.45 & 5.17 \\ \hline D7 & Selene Noir: Dark Icon Pack & Personalization & 0.39 & 0.43 & 4.17 \\ \hline D8 & Runestone Text Editor & Developer Tools & 0.39 & 0.48 & 1.67 \\ \hline D9 & Anonymous Camera & Photo \& Video & 0.40 & 0.44 & 4.23 \\ \hline D10 & Snippet—Code Sharing & Developer Tools & 0.40 & 0.61 & 3.67 \\ \hline D11 & Our Paws & Lifestyle & \(-\)0.04 & 0.34 & 0 \\ \hline D12 & DodgeBlox & Games & \(-\)0.03 & 0.38 & 0 \\ \hline D13 & Midlands Metalheads Radio & Entertainment & \(-\)0.03 & 0.42 & 0 \\ \hline D14 & Antidote COVID & Games & 0.09 & 0.26 & 0 \\ \hline D15 & Brain Breaker & Games & 0.10 & 0.49 & 0.08 \\ \hline C.C. & Correlation Coefficient with H.J. & & 0.937 & 0.399 & - \\ \hline \end{tabular}
\end{table} 
Table 8: SBERT, Doc2 Vec and H.J. similarity scores between app descriptions and query 6 description.