# Selecting Workers Wisely for Crowdsourcing When Copiers and Domain Experts Co-exist
## 7 Experiments
### Experimental Settings  
We conducted experiments on two real-world datasets and one synthetic dataset:  
* MovieLens[63]: This dataset contains 1,000,209 anonymous ratings of approximately 3900 movies made by 6040 MovieLens users. All ratings are in the following format: UserID-MovieID-Rating. UserIDs range between 1 and 6040. MovieIDs range between 1 and 3952. Ratings are made on a 5-star scale. Another file contains the domain to which each movie belongs. Each user has at least 20 ratings;
* Anime ([[https://www.kaggle.com/CooperUnion/anime-recommendations-database](https://www.kaggle.com/CooperUnion/anime-recommendations-database)]([https://www.kaggle.com/CooperUnion/anime-recommendations-database](https://www.kaggle.com/CooperUnion/anime-recommendations-database)) (accessed on 19 December 2021)): This dataset contains information on user preference data from 73,516 users on 12,294 anime works. Each user is able to add anime works to his/her completed list and give it a rating. This dataset is a compilation of those ratings. All data are displayed in the following format: userid-animid-rating (ratings range from 0 to 10). The domain information of anime works is in the description document of the anime works;
* Synthetic dataset: This dataset was synthesized on the basis of MovieLens dataset by manually adding copiers. We added different proportions of copiers, in which each copier randomly copies a worker in the MovieLens dataset. Due to randomness, we generated 100 synthetic datasets for each proportion, and the experimental results of each method were averaged. We discuss the performance of various algorithms via tuning the proportion of copiers. Now, there are many crowdsourcing methods to solve the crowdsourcing problem from different angles. We compared our method with the following seven baseline methods:  
* RandomMV: This method uses a random strategy for task assignment and aggregates workers' answers to generate truth by using majority voting;
* DAS[7]: This method also uses the random strategy for task assignment. For truth discovery, it uses the EM algorithm, which calculates worker accuracy and truth;
* ASKIT1[17]: This method uses an entropy-like method to define the uncertainty of each task and infers the truth by majority voting. The task with the highest uncertainty is the next one to be assigned to the worker;
* CDAS[6]: It provides an estimated accuracy for each result from workers based on the workers' historical performances. Each task we are already confident in is terminated and no longer assigned to workers. At each step, CDAS selects at random a non-terminated task to assign to the incoming worker;
* ARE[14]: This method selects one expert for each task based on the professional domain and proficiency of workers' knowledge. In this model, experts accept tasks equal to or lower than their proficiency;* MDC [15]: This method considers the domain factors of tasks and workers to aggregate better results in the truth discovery stage. Calculate the truth, and update the worker's domain credibility by the proportion of the task in each domain, the worker's answer, and domain credibility;
* SWWC-NoCopier: This is a variant of our method SWWC, but it assumes all workers are independent. We adopted accuracy and efficiency to evaluate the performance of the methods. The goal of crowdsourcing is to obtain the best answer for each task. Therefore, the lower the error obtained by the method and the closer the truth estimated by the method is to the ground truth, the better. Thus, we used the MAE and RMSE to measure accuracy. In addition, we used program running time to measure efficiency. Because of the randomness, we ran each method 100 times to evaluate their average performance:  
* MAE: This quantifies the average error between the estimated truth and the ground truth. The lower the MAE, the better the estimated truth.