# Designing for Hybrid Intelligence: A Taxonomy and Survey of Crowd-Machine Interaction
## 3 Methodological Approach
In fact, this empirical-to-conceptual methodological approach has been a common procedure for data collection in the taxonomy-building activity (e.g., [51; 52; 53]), involving a set of systematic processes that range from a literature search to data filtering and classification. For taxonomic validation, a conventional approach for corpus construction was used as previously described in [54]. Essentially, the sample used in this study is an expanded version of that used in [41]. This was achieved by following a living systematic review protocol [55], where the search strategy is maintained and updated in a continuous manner as new studies become available. For the purpose of this review, a simplistic Boolean query formulation was applied using the following sequence of terms:  
((\"_crowd_'-_AI_\" OR \"_AI-crowd_'-_' OR \"_crowd_'-_machine_\" OR \"_machine-crowd_'-_' OR \"_crowd_'-_computing_\") AND (\"_interact_'\"))  
This study expanded upon a previous corpus to accommodate a new set of possible settings in which crowd-AI interaction occurs. This was done due to two main reasons. First, a more recent picture of the state-of-the-art in this domain was needed. To this end, only papers published in the last five years (2018-2022) as of 17 December 2022 were inspected. Second, most of the studies considered for taxonomy validation in [41] com  
Figure 1: Iterative taxonomy development process flow (**a**) and methodological details underlying the work undertaken in this study (**b**). Adapted from [PERSON] and co-authors [6]. prised human-AI interaction at an individual level, while here, the focus is on evaluating arrangements involving crowds mixed with AI. The present work is also more restrictive in terms of peer-reviewed studies since this contribution only considered journal articles and conference papers. From a systematic search for publications indexed by the Dimensions database, which contains records from diverse digital libraries such as ACM Digital Library, IEEE Xplore, SpringerLink, and Science Direct with large coverage when compared to Web of Science and Scopus [56], content types such as adjunct/companion proceedings, panels, tutorials, book reviews, correspondence articles, introductions to special issues, doctoral colloquiums and student research competitions, keynote talks, commentaries, and course summaries were disregarded to ensure high-quality results. The search returned 593 publication records. After initial scrutiny of the titles and abstracts, along with the removal of papers that did not meet the inclusion criteria, a total of 138 studies were selected for further appraisal. To be eligible for inclusion, studies had to describe original research from primary or secondary literature addressing the broader domain of human-centered AI with a focus on crowd-AI interaction. As can be perceived from Figure 1, this selection resulted in 25 research studies published in English-written, peer-reviewed manuscripts (see Appendix A for details). The final set of papers chosen provided a reliable source of information for testing the taxonomic proposal since they presented a diverse set of scenarios. As an integral part of the iterative taxonomy development process proposed in [6], the meta-characteristic of the taxonomy was determined to be its focus on _functional properties and attributes of hybrid crowd-AI systems_. Through a socio-technical lens grounded on the foundational aspects of crowd computation [57] and its embodiment in hybrid human-AI systems [58], the definition of this meta-characteristic allowed to frame and guide the taxonomy development process until the _subjective ending conditions_ previously mentioned at the level of robustness, comprehensiveness, conciseness, extendibility, and explanatory nature of the taxonomy were fulfilled.