# Selecting Workers Wisely for Crowdsourcing When Copiers and Domain Experts Co-exist
## 3 Problem Definition  
In this section, we formally define the problem of selecting workers wisely for crowdsourcing and provide the details of our crowdsourcing model. The system selecting workers wisely for crowdsourcing generally involves four components in its life cycle:  
**Inputs** include:  
* A set of domains, D. \(D=\{1,2,\ldots|D|\}\). This contains all the possible domains involved in the tasks in the system. All domain IDs are named from \(1\) to \(|D|\);
* A set of tasks, T. Each \(t_{j}\in T\) is a numerical selection task. \(|t_{j}|\) indicates the number of options for the task \(t_{j}\). For example, if the task is to give [PERSON]'s age, then the \(|l|\) is \(130\). In addition, we use \(D_{t_{j}}\) to represent the domain set involved in task \(t_{j}\);
* A set of workers, W. Each \(w_{i}\in W\) applies to complete tasks in the crowdsourcing system;
* A set of labels, L. Each label \(L_{i}\in L\) is the information voluntarily provided by the worker at the time of registration, indicating some characteristics of the worker, such as age, occupation, etc. These labels are used to better initialize the domain expertise of workers in the initialization algorithm;
* M. The upper limit of the number of tasks each worker can complete. We set M as a constant;
* K. The number of workers required for each task. We set K to another constant. **Intermediate variables** are generated and updated during the SWWC procedure:  
* Worker domain expertise vector. Each \(w_{i}\) is modeled as a vector \(\left[v_{i}^{1},v_{i}^{2},\ldots v_{i}^{|D|}\right]\), where each \(v_{i}^{k}\in[0,1]\) indicates the expertise of worker \(w_{i}\) in answering tasks in domain \(k,1\leq k\leq|D|\). A higher value \(v_{i}^{k}\) means that worker \(w_{i}\) is better at domain \(k\). The system updates it after \(w_{i}\) completes tasks;
* Fine-grained worker credibility, \(q_{i}^{j}\). This reflects the capability of worker \(w_{i}\) providing true value to task \(t_{j}\). It is calculated based on the worker domain expertise vector;
* Selected workers set, \(\hat{W}_{t_{j}}\). This collects the workers selected to complete task \(t_{j}\). \(|\hat{W}_{t_{j}}|\leq K\) (Section 4.2);
* \(m_{w_{i}}\). This depicts the number of tasks that has already been assigned to worker \(w_{i}\). \(m_{w_{i}}\) should never exceed M;
* A set of answers provided by workers after task assignment, A. Each \(a_{i}^{j}\in A\) depicts the answer provided by worker \(w_{i}\) on task \(t_{j}\);
* Historical task completion records, H. This collection contains all tasks that have been previously completed by workers. Each \(h\in H\) is modeled as a vector \([w_{i},t_{j},a_{i}^{j},\bar{a}_{j}]\), where \(\bar{a}_{j}\) is the estimated truth of task \(t_{j}\). We denote by \(\phi\) the observation of H.  
**Output** is:  
* Estimated truth, \(\bar{a}_{j}\). This is obtained by integrating the answers of all workers on task \(t_{j}\)  
**Ground truth** is the factual truth for each task \(t_{j}\in T\), denoted as \(\hat{d_{j}}\), which is used to measure the effectiveness of methods. Different from most existing methods, we took the dependence of workers into consideration in order to reduce the impact of copiers. Based on the given historical task completion records, we calculated the probability of copying between each pair of workers. The basis for detecting plagiarism among workers is that if two workers \(w_{i}\) and \(w_{i^{\prime}}\) always share the same wrong answers, there may be a dependence between them. We use \(w_{i}\perp w_{i^{\prime}}\) to represent that workers \(w_{i}\) and \(w_{i^{\prime}}\) are independent and use \(w_{i}\sim w_{i^{\prime}}\) to describe that there is a dependence relation between \(w_{i}\) and \(w_{i^{\prime}}\).