# A System Design Perspective for Business Growth in a Crowdsourced Data Labeling Practice
## 5 Aggregating Results (Solution to RQ5)

Table 8: Hypothetical labeling result with 10 items and the contribution of 3 users. \begin{table}
\begin{tabular}{c c} \hline \hline \multicolumn{2}{c}{**Graphical Representation**} & \multicolumn{2}{c}{**Computation**} \\ \hline \hline  
This result also implies that dominant items have been completed based on the accuracy of the responses of medium-level users to the golden data. That is, dominant complete items \(i_{1}\), \(i_{4}\), and \(i_{4}\) are labeled by a group of users whose average responses to the golden data range between 50% and 80%. User A has responded correctly to all golden data, obtaining the whole accuracy score. User B has provided a wrong answer to \(i_{8}^{*-}\), leading to two right responses out of the three golden ones and an accuracy rate of 66.67%. User C has only labeled two golden items and the response to \(i_{5}^{*+}\) is incorrect, resulting in an accuracy rate of 50%. The average rate of users who contribute to labeling the complete dominant items equals 72.22%. This value implies that 100% of the dominant complete items have been labeled by users with a medium accuracy level. Table 10 describes the results derived relative to the golden items. To determine the consistency level of the golden data, we must first calculate the fraction of users providing consistent responses to the default positive and negative labels. For instance, two users have labeled \(i_{2}^{*+}\) via correct True responses. Hence, the responses of users are fully consistent with the default positive label of \(i_{2}^{*+}\), resulting in a 100% consistency level. Similarly, the corresponding level for both \(i_{5}^{*+}\) and \(i_{8}^{*+}\) is 66.67%, consistent with their default labels. The report delivered to the system administrator should state that 33.33% and 66.67% of the golden items involve a high and medium level of consistency, respectively. Note that there is no golden item with a consistency level below 50%. Moreover, out of 25 labels registered, 17 of them correspond to ordinary items and the 8 remaining answers are allocated to the golden items. In other words, 32% of the labeling is devoted to golden items whereas ordinary data account for 68%. \begin{table}
\begin{tabular}{l l} \hline \hline \multicolumn{1}{c}{**Graphical Representation**} & \multicolumn{1}{c}{**Computation**} \\ \hline True \(-\) based domination \((\%)=\frac{|i_{1}|}{|i_{1}|i_{4}|i_{4}|i_{4}|}=\frac{1}{3}\times 100=33.33\%\) \\ False \(-\) based domination \((\%)=\frac{|i_{6}|}{|i_{4}|i_{4}|i_{4}|i_{4}|}=\frac{2}{3}\times 100=66.67\%\) \\ Report \(-\) based domination \((\%)=\frac{|i_{0}|}{|i_{1}|i_{4}|i_{4}|}=0\times 100=0\%\) \\ \hline \hline \end{tabular}
\end{table} 
Table 9: _Cont._The functionality of the metrics proposed in our system can be compared in detail with those of recent data labeling systems and algorithms. The labeling system designed by [31] introduces a reliability metric associated with the performance of users--while ours provides a bridge between the latter and the quality of the data labeled. These authors also set a minimum reliability threshold below which the responses of the corresponding users are omitted from the final aggregated results. Ref. [32] designed an accuracy metric to evaluate the output results obtained. Their metric was defined in terms of the number of tasks (items) whose estimated labels (aggregated result) were consistent with their True labels (golden ones) divided by the total number of tasks. This definition implies that the accuracy metric in their study does not lead to an entity-based output. The corresponding accuracy metric is, indeed, applied to justify the entire dataset. Conversely, as discussed above, our paper proposes an item-by-item evaluation of accuracy, highlighting its entity-based nature. The data labeling system presented by [16] introduces a metric called precision that specifically measures the accuracy of individual annotations against a gold standard.