# Leveraging Social Network Analysis for Crowdsourced Software Engineering Research
## 3 Data and Methods
### Data Collection  
To extract relevant publications, we used electronic databases and conducted a keyword-based search. Data on CSE publications were collected from the six most popular databases: IEEE Xplore, ACM Digital Library, SpringerLink, ScienceDirect, Web of Science, and Scopus. Only scientific publications were considered.  
The search extracted CSE publications until the end of 2020, with no specified start date. A list of search terms that had been used in previous studies [4] was also used in this research to extract all the publications from the above databases (Table 1). In order to reduce potential bias, two researchers judged the quality of papers and their resulting discussions. Three cycles of review and refinement were carried out by both researchers. After each cycle, both researchers met to discuss the results. The stages of the selection process and numbers of publications found are presented in Table 2.  
\begin{table}
\begin{tabular}{l} \hline (software crowdsourcing) \\ \hline (crowd OR crowdsourcing OR crowdsourced OR crowd-sourced) AND (software engineering) \\ \hline (crowd OR crowdsourcing OR crowdsourced OR crowd-sourced) AND (software development) \\ \hline (crowd OR crowdsourcing OR crowdsourced OR crowd-sourced) AND (software requirements) \\ \hline (crowd OR crowdsourcing OR crowdsourced OR crowd-sourced) AND (software design) \\ \hline (crowd OR crowdsourcing OR crowdsourced OR crowd-sourced) AND (software coding) \\ \hline (crowd OR crowdsourcing OR crowdsourced OR crowd-sourced) AND (software testing) \\ \hline (crowd OR crowdsourcing OR crowdsourced OR crowd-sourced) AND (software verification) \\ \hline (crowd OR crowdsourcing OR crowdsourced OR crowd-sourced) AND (software evolution) \\ \hline (crowd OR crowdsourcing OR crowdsourced OR crowd-sourced) AND (software maintenance) \\ \hline \end{tabular}
\end{table}
Table 1: Search terms.  
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Stage** & **IEEE** & **Scopus** & **SpringerLink** & **Web of Science** & **ACM Digital Library** & **ScienceDirect** & **Total** \\ \hline Retrieved publications & 1120 & 1351 & 79 & 766 & 85 & 29 & 3430 \\ \hline After the screening process & 256 & 375 & 20 & 195 & 65 & 12 & 924 \\ \hline After deleting duplicates & 44 & 201 & 19 & 191 & 28 & 1 & 484 \\ \hline Snowballing & & & +25 & & & 509 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Stages of the data collection process and numbers of publications found.  
We searched the above strings in the fields of titles, abstracts, and index terms. We used Google Scholar to extract the citation count and to record which publications cited which articles so we could determine which articles to use in our citation network. Compared with the other databases, Google Scholar provides greater access to citation information [27] and, therefore, it has been used as a citation reference in a wide range of literature where the creation of citation networks was needed (e.g., [28; 29]).  
The search of the six databases returned 3430 publications. We screened these results by examining the metainformation on the publications and then manually filtered out publications not related to the field of CSE, resulting in 924 publications. Then, we removed duplicated publications that were indexed in multiple databases, resulting in 484 publications.  
Finally, we conducted snowballing to identify any missing publications by searching the references of the included publications. We found 25 new publications and added them to our publication list. This stage resulted in 509 publications, each of which was given an identification (ID) designation (P1, P2, P3,... ). They can be accessed using the link [[https://github.com/salyahya99/CSE-Publications/](https://github.com/salyahya99/CSE-Publications/)]([https://github.com/salyahya99/CSE-Publications/](https://github.com/salyahya99/CSE-Publications/)) (accessed on 1 February 2022).