# Designing for Hybrid Intelligence: A Taxonomy and Survey of Crowd-Machine Interaction
## 1 Introduction and Context  
Crowd-centered design is far from a trivial undertaking, and this is even more challenging when trying to implement hybrid intelligence models incorporating human cognition into algorithmic-crowdsourcing workflows [1]. In fact, crowd-algorithm interaction has recently reached a certain level of maturity, and a vast range of crowd-powered algorithms have successfully been applied in areas like medical image segmentation [2] and games with a purpose (GWAP) [3]. In these instances, crowds of untrained (non-expert) online workers have proved to provide similar results in terms of detection accuracy when compared to other groups such as domain knowledge experts, medical students, and experienced crowd workers. Further investigations in this burgeoning domain have also shown that the use of crowd-algorithm hybrids can outperform crowd-only techniques in accomplishing taskslike examining protein interactions and chemical reactions that are very common in the field of network biology [4]. Nonetheless, the taxonomic rationale behind the mass interaction efforts between crowds and machines as an integrated and complex socio-technical system is not completely understood, and there is a need to find novel ways of characterizing this body of work in its whole range. To mitigate this brittleness, a review of the main activities and contexts in which such crowd-AI ensembles have been investigated was carried out to develop a taxonomic scheme as comprehensive as possible to capture the nuances that are unique in comparison with other types of interactions between humans and computational systems. For more than three decades, taxonomy development has been seen as a crucial part of socio-technical research within the field of CSCW [5]. To some extent, taxonomies provide a useful guide and theoretical foundation for assessing technological developments due to their capability to organize complex concepts and knowledge structures into understandable formats [6]. By going back in the course of time, one may find several taxonomic approaches that formed the basis for the understanding of the task types that are currently present in many crowdsourcing systems. For a review of prior taxonomic proposals, the reader is referred to [PERSON] and co-authors [7]. In retrospect, [PERSON] [8] proposed a circumplex model of group tasks intended to characterize their nature (e.g., decision-making) into four quadrants that reflect the processes involved in their execution (i.e., generate, choose, negotiate, and execute). When moving even further back in history, [PERSON] [9] asserted the importance of aspects like task difficulty and intrinsic interest which are seen as foundational in several conceptual frameworks proposed to characterize the broader crowdsourcing phenomena (e.g., [10; 11]). According to some authors, [PERSON]'s [12] time-space matrix is a landmark in the field of CSCW and inspired the development of descriptive models such as the Model of Coordinated Action (MoCA) [13], which frames each collaborative work arrangement on a continuum of synchronicity (synchronous vs. asynchronous), physical distribution, scale (i.e., number of participants), number of communities of practice involved, nascence and planned permanence of coordinated actions, and turnover. More recently, [PERSON] and colleagues [14] executed a set of data collection and processing procedures involving structured interviews in order to create a taxonomic scheme covering the components related to the collaboration technology support in home care work, while other authors have devoted most of their efforts to the design of innovative taxonomic interfaces [15]. In addition, there is now an emerging body of research documenting the different levels of hybrid intelligence in human-algorithm interactions.