# A System Design Perspective for Business Growth in a Crowdsourced Data Labeling Practice
## 5 Aggregating Results (Solution to RQ5)
\begin{table}
\begin{tabular}{c c c} \hline \hline \multicolumn{2}{c}{**Status of**} & \\ \multicolumn{2}{c}{**Ordinary Items**} & \\ \hline \multirow{5}{*}{Complete items} & \(\bullet\) & Contribution of complete items with dominant labels: \\  & & Domination \((\%)=\frac{\text{Count of complete ordinary items with dominant result}}{\text{Total number of complete ordinary items}}\times 100\) \\  & & Contribution of complete items with semi-dominant labels: \\  & & Semi – domination \((\%)=\frac{\text{Count of complete ordinary items with semi–dominant result}}{\text{Total number of complete ordinary items}}\times 100\) \\  & & Contribution of complete items with non-dominant labels: \\  & & Non – domination \((\%)=\frac{\text{Count of complete ordinary items with non–dominant result}}{\text{Total number of complete ordinary items}}\times 100\) \\ \hline \multirow{5}{*}{Complete items} & \(\bullet\) & Contribution of complete dominant items with True response: \\  & & True – based domination \((\%)=\frac{\text{Count of complete dominant ordinary items with True response}}{\text{Total number of complete dominant ordinary items}}\times 100\) \\  & & Contribution of complete dominant items with False response: \\  & & False – based domination \((\%)=\frac{\text{Count of complete dominant ordinary items with False response}}{\text{Total number of complete dominant ordinary items}}\times 100\) \\  & & Contribution of complete dominant items with Report response: \\  & & Report – based domination \((\%)=\frac{\text{Count of complete dominant ordinary items with Report response}}{\text{Total number of complete dominant ordinary items}}\times 100\) \\ \hline \multirow{5}{*}{Incomplete items} & \(\bullet\) & Contribution of incomplete items with no label: \\  & & No label incompletion \((\%)=\frac{\text{Count of incomplete ordinary items with no label}}{\text{Total number of incomplete ordinary items}}\times 100\) \\  & & Contribution of incomplete items with at least one single label: \\  & & Min label incompletion \((\%)=\frac{\text{Count of incomplete ordinary items with at least one single label}}{\text{Total number of incomplete ordinary items}}\times 100\) \\ \hline \multirow{5}{*}{Accuracy of user responses} & \(\bullet\) & Contribution of users with high accuracy rate in labeling complete dominant items: \\  & & High – level users contribution \((\%)=\frac{\text{Count of completed dominant ordinary items by highly accurate users}}{\text{Total number of complete dominant ordinary items}}\times 100\) \\  & & Contribution of users with medium accuracy rate in labeling complete dominant items: \\  & & Medium – level users contribution \((\%)=\) \\  & & _Count of completed dominant ordinary items by medium accurate users_ \\  & & _Total number of complete dominant ordinary items_ \\  & & \(\bullet\) Contribution of users with low accuracy rate in labeling complete dominant items: \\  & & Low – level users contribution \((\%)=\frac{\text{Count of completed dominant ordinary items by loudy accurate users}}{\text{Total number of complete dominant ordinary items}}\times 100\) \\  & & _The accuracy rate of each user is computed as follows:_ \\  & & Accuracy rate of User \((\%)=\frac{\text{Count of golden items correctly labelled}}{\text{Total number of golden items labelled}}\times 100\) \\  & & Average accuracy rate of Users that labeled a dominant item \((\%)=\) \\  & & _Sum of accuracy rates of Users that labeled a dominant item_ \\  & & _Total number of Users labelling a dominant item_ \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Cont. We must also define a reliability criterion for the aggregated data. The criterion proposed maps the accuracy of the responses of contributors to golden data into the complete dominant items. In particular, it specifies the contribution percentage of highly, moderately, and lowly accurate users in the answers submitted to dominant items.