# Designing for Hybrid Intelligence: A Taxonomy and Survey of Crowd-Machine Interaction
## 5 Validation and Assessment of the Proposed Taxonomy
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**ID** & **Experimental Settings** & **Pre-Selection Mechanism(s)** & **Cost per HIT and Platform(s)** & **Time Allotted** \\ \hline \multirow{3}{*}{P1} & 5-month-long deployment and testing & & \multicolumn{2}{c}{\(\$0.142\) (Phase-1 deployment);} & \multirow{3}{*}{\(\sim\)10 min (per} \\  & with real users (\(n=80\) crowd workers) & & & MTurk; Hangoutsbot \\ \hline \multirow{3}{*}{P2} & Ensemble method combining multiple & & \multicolumn{2}{c}{\(\$0.90\) (Segmentation);} & \multirow{3}{*}{\(142.6\) s (per frame} \\  & results on individual frame & & \(\$0.15\) (Scribble);} & \\  & segmentations and crowd-based & & \(\$0.15\) (Scribble);} & \\  & propagated segmentation results & & MTurk & method scribbles) \\  & (\(n=70\) crowd workers) & & & \\ \hline \multirow{3}{*}{P3} & 4-week testing (\(n=17\) participants), with an unspecified number of crowd workers & \(>\)95\% assignment approval & \(\sim\)510/hour (\(\$0.02\) for each task & \(\sim\)3 s (per labeled \\  & with an unspecified number of crowd workers & rate; Gold standard question & performed on MTurk) & question sensor \\ \hline \multirow{3}{*}{P5} & Classification of potential studies for a systematic literature review (\(n=147\) & \(>\)70\% overall accuracy; Worker screening based on two & \(\$10\)/hour; & \\  & & & & MTurk & - \\  & crowd workers) & test questions & & & \\ \hline \multirow{3}{*}{P6} & Purpose-mechanism annotation & \(>\)95\% acceptance rate; & \(\$30\)/hour (Upwork-worker 1); & 1.3 min (per document \\  & analogical search (\(n=3\) crowd workers & Training step based on a gold & \(\$20\)/hour (Upwork-worker 2);} & \multicolumn{2}{c}{amotation); 4 min} \\  & per document), with an unspecified number of crowd workers & standard example before the task execution & \(\$10\)/hour (\(\$0.70\) for each task & (overall task \\  & number of crowd workers & task execution & performed on MTurk) & completion) \\ \hline \multirow{3}{*}{P9} & Contextual bandit algorithm and agent & \multirow{3}{*}{\(\$12\)/hour;} & \multirow{3}{*}{-} \\  & deployment powered by AI-based & & & \\  & request strategies for visual question & & & MTurk & - \\  & answering, with an unspecified number of crowd workers & & & \\ \hline \multirow{3}{*}{P13} & Performance evaluation of a crowd-AI & \multirow{3}{*}{\(>\)95\% overall task approval} & \(\$0.20\) for each worker & \multirow{3}{*}{-} \\  & hybrid framework through real-world & & & \\  & datasets (\(n=3\) crowd workers per image in a crowd query) with an unspecified number of crowd workers & rate; \(>\)-1000 HITs completed & & \\ \hline \multirow{3}{*}{P14} & A method for AI worker evaluation & \multirow{3}{*}{No strategies were deployed to target malicious workers} & \multirow{3}{*}{\(240\)s for 2 h of labor;} & \multirow{3}{*}{-} \\  & that uses a “divide-and-conquer” & & & \\ \cline{1-1}  & strategy for dynamic task assignment & & & MTurk & - \\ \cline{1-1}  & with an unspecified number of crowd workers & & & \\ \hline \multirow{3}{*}{P16} & Evaluation of hybrid & \multirow{3}{*}{\(>\)92\% approval rate; \(>\)500 HITs} & \(\$9\)/hour (\(\$0.20\) for each task & \(\$0\) s (per HIT \\ \cline{1-1}  & image annotation based on time & completed & performed on MTurk) & completion) \\ \cline{1-1}  & completion and quality, with an unspecified number of crowd workers & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Methodological remarks extracted from primary studies reporting paid crowdsourcing experiments conducted remotely. Regarding the filtering mechanisms used for early pre-selection of crowd workers, the review of the literature showed five studies where the HIT acceptance rate was set to more than 95%. Moreover, this contribution also identified four studies where the number of tasks completed by a potential crowd worker had to be at least 1000. From this scoping review, a total of five experiments involved some type of ground truth in the form of a gold standard or test question.