# Designing for Hybrid Intelligence: A Taxonomy and Survey of Crowd-Machine Interaction
## 5 Validation and Assessment of the Proposed Taxonomy
As such, the first taxonomic unit contains the spatio-temporal elements (T1) that frame crowd-AI interaction in relation to the original time-space matrix proposed by [PERSON] [12]. In brief terms, this classification model categorizes interactions as follows: same place/time,  
Figure 4: Example of a taxonomic scheme used to classify a crowd-AI interaction scenario [39]. different places/same time, same place/different times, or different places/different times. To a broad extent, crowd-AI interactions can occur in asynchronous or real-time settings where the individuals that constitute the crowd can be physically and virtually co-located or geographically dispersed (remote). In addition, the worker location and task duration time [11] were also considered, as the latter is intimately connected to the time frame or limit that is set to complete a task. In the example provided in Figure 4, a nearly real-time on-demand crowd-powered system is proposed to collect responses from crowd workers that can be at any location but need to be available to provide contributions in real-time due to the quickly changing contextual requirements underlying the type of tasks performed. Looking at the results of the taxonomy-based literature review in detail, a total of 84% (\(n=21\)) of included papers have reported temporal and/or spatial aspects of crowd activity. As a brief example, [PERSON] and colleagues [119] introduced a mixed-initiative system with an annotation time of 1 min per paper on average in analogy matching tasks. In terms of real-time crowd-AI settings, some primary studies (e.g., [36; 39; 40; 98; 120]) presented synchronous interactions between crowd members, although most of the crowdsourcing systems relied on an asynchronous model. Consistent with the previous literature, the most addressed taxonomic unit is related to task design, assignment, and execution (T2), with a total of 25 primary studies. In crowdsourcing experiments, task design is seen as a cornerstone to achieving the goals of a project or campaign since the characteristics and configuration of crowdsourced tasks influence the general outcomes obtained from the crowd [91]. In general, different types of tasks were found in the selected sample. As mentioned before, tasks differ both in terms of attributes, complexity, and granularity [11]. For instance, Scalpel-CD [121] generates label inspection microtasks in a dynamic way, while Evorus [39] focuses on classification tasks in the form of voting. A slightly different task specification is employed in Photo Sleuth [38], where crowd workers are invited to perform person identification/recognition tasks that are therefore augmented with visual tags to allow portrait seeking. Moreover, CollabLearn [36] is based on crowd query tasks where human processing is needed to highlight damaged areas from cultural heritage imagery. A somewhat related body of work (e.g., [34]) has sought to support the execution of crowd-in-the-loop interactive image labeling tasks with the ultimate goal of enhancing AI-powered damage scene assessment algorithms. All in all, the task-related aspects discussed in the growing literature on the interplay between crowdsourcing and AI systems have been playing an indispensable role in explaining complex relationships among crowd inputs and further integration into hybrid workflows. Extrapolating to the ethical principles and standards in crowd-AI settings (T3), the review only identified nine papers (36%) that explicitly discuss ethical behaviors from a requester-, crowd- or even AI-centered standpoint. Despite the recognized need for fair payment and long-term career building in online crowd work platforms [122], this study shows that the ethical concerns underlying the interaction-centric crowd-AI activity are often overlooked from a practical perspective, despite some examples of strategies presented in the crowdsourcing literature such as ensuring fair compensation by paying crowd workers in conformity with the complexity of the task being performed [123].