# Designing for Hybrid Intelligence: A Taxonomy and Survey of Crowd-Machine Interaction
## 5 Validation and Assessment of the Proposed Taxonomy  
This study proposes a taxonomic framework aimed at accommodating a diverse set of infrastructurally supported crowd-algorithm interactions that occur in a certain time and space within two separate orders of intelligence, which, therefore, can be combined in a hybrid model architecture. The interactions occurring in this hybrid space have a set of unique contextual and situational aspects and must be guided by ethical guidelines, rules, and principles in order to combine crowd and machine workflows effectively and transparently. To validate the proposed taxonomy and demonstrate its utility, this contribution examined the applicability of the taxonomy in a total of twenty-five studies presenting some type of crowd-machine interaction. This is in line with the need for a methodologically rigorous inspection of the possible effects of hybrid intelligence in practical settings. For instance, substantial literature on human-AI interaction has developed quickly across different areas [116], but few attempts have been made to gather evidence about this intersectional space at a crowd scale and thus understand the uses and limitations of hybrid crowd-AI systems from a socio-technical design viewpoint. The results of the taxonomy-based review are provided in Figure 3, accompanied by an example of a scheme used to explain the rationale behind the taxonomic classification (Figure 4). Further details regarding the 14 journal articles and 11 conference papers selected for taxonomy-based literature analysis are given in Tables A1 and A2 in the Appendix A. In order to determine whether each category of the taxonomy was either present or absent, the following levels were considered:  
Figure 3: Synthesis of the literature analysis based on the taxonomy proposed. Fully addressed: The manuscript clearly emphasizes the specific elements underlying the taxonomic category by addressing one or more of its unique attributes, with a potential experiment, solution, or case study demonstrating applicability. For instance, [PERSON] and co-authors [38] make explicit reference to the contextual information (e.g., biographical details) provided to the user about each portrait in Photo Sleuth, a crowd-AI-enabled face recognition platform where a crowd of both expert and non-expert volunteers can tag a picture using this supplementary piece of contextual data to aid the decision process. Not addressed: The work does not directly address any of the aspects that are inherent to the category under consideration. Partially addressed: The study provides details that can be used to address the particular taxonomic category, even if not explicitly mentioned in the manuscript. By way of example, [PERSON] et al. [117] do not directly provide details about the contextual information required in the natural disaster response setting used for demonstrating the proposed method, but the situational awareness and subsequent timely information required to manage the rapidly evolving scenarios toward well-informed and up-to-date decision-making are implicitly stated. On the basis of insights from previous analytical work, this taxonomically grounded literature review process has been adopted in areas like business intelligence and analytics [118] as a way of iteratively developing and refining taxonomic dimensions and characteristics while pinpointing areas requiring further investigation. As can be seen from Figure 3, the taxonomy presented in this article is far from comprehensive enough to accommodate all types of possible scenarios involving crowd-AI interaction. Instead, the goal is to facilitate a cohesive understanding as a basis for further scrutiny of crowd-computing hybrids in real-world applicative contexts. Note that there are some categories that can co-exist, taking into account the specificity of each situation or use case.