# Selecting Workers Wisely for Crowdsourcing When Copiers and Domain Experts Co-exist
## 7 Experiments
### Comparative Study on Two Real-World Datasets  
We compared SWWC with the baseline approaches mentioned in Section 7.1 with the same set of tasks for qualification. Table 1 and Figure 2 show the performance of these approaches on accuracy and efficiency. We can see that our method performed better than others. Since these datasets do not contain copiers, the MAE of our method SWWC-NoCopier was the lowest, followed by our another method, SWWC, which considers copying. Although there are no copiers in these datasets, SWWC may calculate some workers as copiers in the calculation process, which will cause some misjudgment. However, we can see from the results that there were few misjudgments and little impact on the results. Our method SWWC was still superior to all other baseline methods (except SWWC-NoCopier). \begin{table}
\begin{tabular}{c c c c} \hline \hline  & \multicolumn{3}{c}{**MovieLens**} \\ \hline  & **MAE** & **RMSE** & **Time (s)** \\ \hline SWWC & 0.3677 & 0.7583 & 1855.29 \\ SWWC-NoCopier & **0.3606** & **0.7222** & **14.07** \\ RandomMV & 0.6914 & 1.0969 & 56.31 \\ Dc6S & 0.4364 & 0.7942 & 54.74 \\ ASKIT! & 0.7348 & 1.1110 & 1561.88 \\ CDAS & 0.4134 & 0.7801 & 59.66 \\ ARE & 0.8792 & 1.2247 & 69.53 \\ MDC & 0.3851 & 0.7243 & 69.60 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Comparison of different methods on two real-world datasets (1). RandomMV, ASKIT!, and ARE performed worst on both datasets. The performance of RandomMV largely depends on the quality of the selected worker set: if the set contains more good workers, the accuracy would be high; otherwise, the accuracy would be low. The main problem of method ASKIT is the sequence of task assignment. This method does not focus on selecting high-quality workers for each task. Although the ARE method considers domain factors to conduct task assignment, it only selects one for each task. When no qualified expert was available in the system, ARE failed to assign the related tasks. Therefore, the experimental results of RandomMV, ASKIT!, and ARE were not ideal. CDAS and MDC were second only to our method. CDAS considers the confidence of the estimated values. This is an effective way to improve the accuracy of the results. MDC considers domain factors in the truth discovery process, which is helpful to improve the accuracy. However, they do not consider the domain in the task assignment stage, which resulted in a lower accuracy compared with our method. D&S had a certain effect on improving the accuracy of the results. The EM algorithm enhanced the accuracy in the truth discovery stage, so the deviation between the estimated value and the truth was slight, e.g., a small RMSE. However, the lack of consideration of the domain factor makes it perform poorly in the MAE. In order to verify whether only the domain information type is beneficial to the results, we also experimented with the domain information of broadcast mode on the anime dataset. Anime (Domain 1) in Table 2 is the type of anime, and Anime (Domain 2) is the broadcast mode of anime. Anime (Domain 1) includes Action, Adventure, Comedy, Drama, etc. Anime (Domain 2) includes TV, Movie, OVA, ONA, and Special. From the experimental results, we can see that the MAE was the lowest whether considering the domain type or broadcast mode, which proves our idea: considering the domain factor in task assignment is beneficial to the truth. However, considering different domains, the improvement effect of the results was also different. In our experiment, type was better than broadcast. We measured the efficiency of compared methods by running time. From Tables 1 and 2, we can see that our method SWWC took a relatively long time for conducting copying detection. We iteratively detected copiers while assigning tasks to proper workers, so as to improve the accuracy of crowdsourcing. The variant of our method, i.e., SWWC-NoCopier, that excludes the copier detection module consumed the least execution time. It demonstrated that the efficiency of our method is linear with the number of tasks.