# A System Design Perspective for Business Growth in a Crowdsourced Data Labeling Practice
## 5 Aggregating Results (Solution to RQ5)
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Status of Golden Items** & & **Metric** \\ \hline \multirow{7}{*}{Consistency level of responses} & \(\bullet\) & Contribution of items with a high level of consistency: \\  & & High level of consistency \((\%)=\frac{\text{Count of golden items labeled with a high level of consistency}}{\text{Total number of golden items}}\times 100\) \\  & \(\bullet\) & Contribution of items with a medium level of consistency: \\  & & Medium level of consistency \((\%)=\frac{\text{Count of golden items labeled with a medium level of consistency}}{\text{Total number of golden items}}\times 100\) \\  & \(\bullet\) & Contribution of items with a low level of consistency: \\  & & Low level of consistency \((\%)=\frac{\text{Count of golden items labeled with a low level of consistency}}{\text{Total number of golden items}}\times 100\) \\  & & _The consistency level of each golden item is computed as follows_: \\  & & Consistency level of golden item \((\%)=\frac{\text{Count of answers consistent with the predefined label of the golden item}}{\text{Total number of answers submitted to the golden item}}\) \\ \hline \multirow{7}{*}{Percentage of responses’ type} & \(\bullet\) & Contribution of golden responses among all labeled items: \\  & & Golden labels \((\%)=\frac{\text{Count of golden items labeled}}{\text{Total number of items labeled}}\times 100\) \\ \cline{1-1}  & \(\bullet\) & Contribution of non-golden (ordinary) responses among all labeled items: \\ \cline{1-1}  & & Non – golden labels \((\%)=\frac{\text{Count of non-golden items labeled}}{\text{Total number of items labeled}}\times 100\) \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Proposed metrics to evaluate the status of golden items. item basis. This approach ensures that the output of each completed item can be identified and categorized as low, medium, or high quality. If the administrator is not satisfied with the quality level of certain items, the system allows for the corresponding items to receive further labels until the desired quality is reached. In this way, overall improvements are guaranteed for the labeled dataset by focusing on the individual measurement of the accuracy of the items' labels. Table 8 describes the labeling result of a hypothetical dataset including 10 images. The images are listed \(i_{1}\) through \(i_{10}\) where the second (\(i_{2}^{*+}\)), fifth (\(i_{5}^{*+}\)), and eighth (\(i_{6}^{*-}\)) ones are positive, positive, and negative golden types, respectively. In this case, the correct label of the positive (negative) golden items is True (False). A, B, and C are three users contributing to the labeling practice. The possible responses are True (T), False (F), and Report (R) with three labels required per item. If doubtful about the correct label, users can skip (S) the images. For instance, user A has skipped the image while both B and C have assigned it a True label. Table 9 graphically and numerically represents the metrics related to the ordinary items. For instance, to compute the _No label incompletion metric_, the denominator enumerates the items that do not receive the three pre-defined labels, i.e., two items, including \(i_{9}\) and \(i_{10}\). Actually, \(i\) has no labels assigned since all users have skipped it and \(i_{10}\) is incomplete since it has only been labeled by B and C and skipped by A. The numerator is defined by the number of items that do not have a label, i.e., item \(i_{9}\). Thus, the _No label incompleteness rate_ is \(\frac{1}{2}\), which means that 50% of the incomplete items have no label. \begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**User**} & \multicolumn{8}{c}{**Item**} \\ \cline{2-11}  & \(i_{1}\) & \(i_{2}^{*}\) & \(i_{3}\) & \(i_{4}\) & \(i_{5}^{*}\) & \(i_{6}\) & \(i_{7}\) & \(i_{8}^{*}\) & \(i_{9}\) & \(i_{10}\) \\ \hline A & T & T & T & F & T & F & R & F & S & S \\ B & T & T & F & F & T & F & T & T & S & T \\ C & T & S & F & F & F & F & F & F & S & T \\ \hline \hline \end{tabular}
\end{table}