# Despite for SMEs: A Systematic Review of Socio-Technical Cybersecurity Metrics
## 3 Systematic Review Methodology  
We performed a systematic literature review to address our research questions. To ensure broad coverage of the cybersecurity metrics field, we employed a novel systematic review methodology blending active learning and snowballing (SYMBALS, [58]), which combines existing methods into a swift and accessible methodology, while following authoritative systematic review guidelines [59; 60; 61]. Active learning is one of the cornerstones of the SYMBALS approach. Active learning is commonly applied in the title and abstract screening phase of systematic reviews, where researchers start with a large set of papers and prefer not to screen them all manually [62]. Active learning is uniquely suited to this task, as this machine learning method selects the ideal data points for an algorithm to learn from. SYMBALS complements active learning with backward snowballing. From a set of included papers, a researcher can find additional relevant papers by consulting references (backward snowballing) and citations (forward snowballing) [63]. Snowballing has proven to be a valuable addition to systematic reviews, even when reviews already include an extensive database search [64]. Backward snowballing is especially useful in uncovering older relevant research. Forward snowballing is not employed within SYMBALS, based on the observation that databases generally have excellent coverage of recent peer-reviewed research. After the development and evaluation of a systematic review protocol for this research, we commenced with the database search step of SYMBALS. We retrieved research from abstract databases (Scopus, Web of Science) and full-text databases (ACM Digital Library, IEEE Xplore, PubMed Central). The Scopus API was used to retrieve an initial set of relevant research. Results from other sources were then successively added to this set. The order in which sources were consulted can be surmised from Table 3. The Python Scopus API wrapper \"pybliometrics\" [65] was used to retrieve all research available through the Scopus API that satisfied the query:  
```
AUHTKEY((security*ORcyber*)

AND(assess*ORevaluate*ORmeasur*ORmetric*ORmodel*ORrisk*ORscore*))ANDANGJAGE(english)ANDDOCTYPE(arORbkORchORcpORcrORre)
```  
The \"AUTHTKEY\" field corresponds to the keywords that authors provided for a paper. Our search query is intentionally broad, as the SYMBALS methodology allows us to deal with larger quantities of research, and we aim to exclude as little relevant research as possible at this stage. We did choose to only include English language research and document types where extensive and verifiable motivations for findings can be reported. Table 3 summarises the query results. ACM Digital Library and IEEE Xplore limit the number of accessible papers to 2000. This means only the 2000 most relevant papers from these sources could be considered. Moreover, IEEE Xplore only allows the use of six wildcards in the search query. We removed the \"security\" and \"cyber\" wildcards for the IEEE Xplore search to comply with this limitation. Any research without an abstract was excluded, as this is vital to the active learning phase of SYMBALS. This led to a small set of exclusions from the PubMed Central database. Duplicate removal was performed based on the research title, although we found that this process was not perfect, due to different character sets being accepted in different databases. Altogether, our dataset resulting from database search comprised 25,773 papers. This exemplifies the broad scope of our research, as the largest initial set of papers from the reviews in Table 1 comprised 4818 papers [43]. The set of 25,773 papers is too large to perform data extraction directly. This is where the active learning phase of SYMBALS comes in. We chose to use ASReview in this phase, a tool that offers active learning capabilities for systematic reviews, specifically for the title and abstract screening step [62]. Many other active learning tools exist that are worth considering [66].