# Designing for Hybrid Intelligence: A Taxonomy and Survey of Crowd-Machine Interaction
## 5 Validation and Assessment of the Proposed Taxonomy
\begin{table}
\begin{tabular}{c c c c c} \hline \hline \multicolumn{1}{c}{**ID**} & \multicolumn{1}{c}{**Experimental Settings**} & \multicolumn{1}{c}{**Pre-Selection**} & \multicolumn{1}{c}{**Cost per HIT and**} & \multicolumn{1}{c}{**Time Allotted**} \\ \hline \multirow{4}{*}{P17} & Evaluation of crowd responses and \multicolumn{1}{c}{computational performance in} & \multirow{2}{*}{\(>\)95\% overall task approval rate; \(>\)=1000 HITs completed} & \multirow{2}{*}{\(\$\$\)0.05 for each worker per time classification; MTurk} & 0.0227 (average time taken to accomplish each streaming urban monitoring task using a hybrid crowd-AI model) \\  & & & & \\ \hline \multirow{4}{*}{P18} & Evaluation of model performance to \multicolumn{1}{c}{re-query or not crowdsourced} & \multicolumn{1}{c}{identifying inattentive workers; Annotators with more than 15\% incorrect annotations were disregarded} & \multicolumn{1}{c}{-\$12/hour ($0.06 for each bounding-box annotation); MTurk} & \multirow{4}{*}{-} \\  & & & & \\ \hline \multirow{4}{*}{P19} & Randomized online experiments & \multirow{4}{*}{-} & \$7.28/hour plus bonus & \multirow{4}{*}{} \\  & comparing the performance of a & & payments of 20\% to the top & \\  & computer vision model and a crowd of & & participants; Experiment & \\  & 15,016 individuals in tasks related to & \multirow{4}{*}{-} & hosted on an external website & 15 min (per task completion) \\  & the detection of authentic vs. deepfake & & (i.e., Detect Fakes); & \\  & videos (\(n=5524\) participants: & & 304 participants recruited & \\  & Experiment 1; \(n=9492\) participants: & & from Prolific & \\  & Experiment 2) & & & \\ \hline \multirow{4}{*}{P20} & Performance evaluation of a dynamic & \multirow{4}{*}{\(>\)95\% overall task approval} & \multirow{4}{*}{\(\$\)0.20 for each crowd worker rate; \(>\)=1000 HITs completed} & \multirow{4}{*}{\(\$\)0.020 for each crowd worker per time labeling; MTurk} & 0.0198 s (average time query frequency); 0.0201 s (average time with varying numbers of crowd workers) \\ \cline{1-1} \cline{3-3}  & Evaluation of a hybrid framework & & & 0.0198 s (average time of \\  & combining expert and crowd & & & \\   intelligence with explainable AI for & \multirow{4}{*}{\(>\)=95\% task acceptance rate} & \multirow{4}{*}{Unspecified amount above the minimum requirement on MTurk ($0.01 per assignment)} & \multirow{4}{*}{\(\$\)0.0198 s (average time of \\  & (\(n=3\) crowd workers per HIT plus & & & \\  & 5 experts), with an unspecified number & & & \\  & of crowd workers & & & \\ \hline \multirow{4}{*}{P25} & Development of a crowd-AI system for \multicolumn{1}{c}{optimizing smart urban sensing applications (\(n=3\) to 7 crowd workers per task), with an unspecified number of crowd workers} & \multirow{4}{*}{\(>\)95\% overall task approval rate; \(>\)=1000 HITs completed} & \multirow{4}{*}{\(\$\)0.05 for each crowd worker per time classification; MTurk} & \multirow{4}{*}{-} \\ \cline{1-1} \cline{3-3}  & & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: _Cont._average times of HIT completion in accordance with the complexity and requirements of each task, while a remarkable number of primary studies (\(n=10\)) did not mention the total number of crowd workers involved in the experiment. Nonetheless, some studies involve both crowd workers and experts in their experimental settings, with a crowd size ranging from 2 to 7 crowd workers per task and a maximum size of 147 paid online workers in a single experiment.