# A System Design Perspective for Business Growth in a Crowdsourced Data Labeling Practice
## 5 Aggregating Results (Solution to RQ5)  
RQ5 involves retrieving the outputs of the labeling system and delivering managerial reports. In this regard, the ultimate goal of the system consists in providing feedback on the outputs obtained from a labeling practice. This process encompasses the following tasks: aggregating the responses received per item, measuring the reliability of answers, assessing the volume of remaining work, and validating the accuracy of golden items as well as the reports describing the financial performance of users. The aggregation of labeling results will be tackled by separating the analyses related to ordinary and golden items. The answers to ordinary items are unknown to the administrator while golden questions are assigned default labels. Therefore, the analysis of ordinary items should aim at deriving dominant responses under different aggregation rules and reliability metrics. The reason for double checking the responses to golden items is to evaluate the accuracy of their default labels. The answers to golden items are meticulously defined under the supervision of the administrator. Despite this fact, incorrect default labels could still be assigned. Tables 6 and 7 provide a set of metrics for extracting information regarding the status of annotated items within a labeling system. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Status of** & & & **Metric** \\
**Ordinary Items** & & & \\ \hline \multirow{4}{*}{
\begin{tabular}{c} Complete vs. \\ incomplete items \\ \end{tabular} } & \(\bullet\) & Contribution of items with complete labels: \\  & & Completion \((\%)=\frac{Count\ of\ complete\ ordinary\ items}{Total\ number\ of\ ordinary\ items}\) \(\times\) 100 \\  & & Contribution of items with incomplete labels: \\  & & Incompletion \((\%)=\frac{Count\ of\ incomplete\ ordinary\ items}{Total\ number\ of\ ordinary\ items}\times\) 100 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Proposed metrics to evaluate the status of ordinary items. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{\(\mathcal{S}_{Incorrect}^{\star,\
u}\)} & \multicolumn{6}{c}{\(Bonus_{Incorrect}\)} \\ \cline{2-9}  & **0** & \(T^{\textbf{0.2}}\) & \(T^{\textbf{0.5}}\) & \(T\) & \(T^{\textbf{2}}\) & \(T^{\textbf{3}}\) & \(T^{\textbf{10}}\) \\ \cline{2-9}  & \multicolumn{6}{c}{**Relative Penalty Rate (\%)**} \\ \hline
0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\
25 & 100 & 14.13 & 31.66 & 53.3 & 78.19 & 89.82 & 99.95 \\
50 & 100 & 26.26 & 53.3 & 78.19 & 95.24 & 98.96 & 100 \\
75 & 100 & 36.67 & 68.09 & 89.82 & 98.96 & 99.89 & 100 \\
100 & 100 & 45.26 & 78.19 & 95.24 & 99.77 & 99.99 & 100 \\
125 & 100 & 53.3 & 85.1 & 97.78 & 99.95 & 100 & 100 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Relative penalty rate relative to the independent incorrect label case. The status of ordinary items can be analyzed using different metrics. It is crucial to assess the percentage of ordinary data that have received the required number of labels, defined as complete items. This implies differentiating between the contribution of items with dominant, semi-dominant, and non-dominated results. If there is a threshold of three labels per item, three synchronous answers (e.g., True or False) constitute a dominant result, showing a strong consensus of users over the corresponding item. In the case of two similar labels (e.g., True) and one opposite label (e.g., False), the system assigns a semi-dominant state to the item. The non-dominant status occurs when the three users report the item or provide distinct responses (i.e., True, False, and Report). The complete items with a dominant result will display True, False, or Report labels. Among the incomplete ones, the rate of unlabeled items, as well as cases with at least a single answer submitted, will convey useful information to the admin.