## 2 Background
### Crowd Selection in Crowdsourcing
#### 2.1.1 Basic Screening-Based Crowd Selection  
Screening-based crowd selection applies screening methods to select an appropriate crowd for a crowdsourcing task [12]. For selecting a suitable subset of the crowd, screening can be applied at different stages prior to selecting the crowd [13] or during the task [14]. These two methods can be denoted as pre-screening methods and in-task screening methods, and both can contribute to selecting a suitable subset of an online crowd. Pre-screening includes several forms mainly revolving around using filtering, qualification tests and gold standard questions [15]. Using built-in filters supported by the crowdsourcing platform is one of the simplest methods to select crowd members based on explicit basic attributes such as geographical location, age and rating that are usually available in a crowd worker's profile [11; 15]. [PERSON] et al. [16] have found that there is a strong correlation between the geographical location of the crowd and the quality of their work. This is due to language barriers that crowd workers might confront when working on a task that requires language skills. In addition, other reputation-based attributes [17] such as acceptance rate of crowd worker's contributions and the number of previously completed tasks have been utilized when selecting a reliable crowd for crowdsourcing [18]. A plethora of studies have utilized the acceptance rate and previously completed tasks as constraints for crowdsourcing microtasks such as gathering judgments [19] and crowdsourcing testing [20]. Regardless of the simplicity associated with filtering methods, they can enhance the quality of the work produced by the crowd.  
Another pre-screening method is using qualification tests to narrow the pool of crowd workers to those possessing qualifications suitable for a crowdsourced task. Qualification tests are a superior kind of filter and consist of questions that can reflect a crowd worker's ability in performing a task. If the crowd worker passes the test, he becomes eligible to enter the task, otherwise he is eliminated [12]. Indeed, they have been utilized by large number of studies when selecting a subset of qualified crowd workers. Qualification tests are usually task-specific [17], meaning that studies design qualifications tests tailored for the task they are crowdsourcing to best capture the qualifications and abilities.  
[PERSON] et al. [21] have designed qualification tests for image annotation and have found that they increase the chance of obtaining a high-quality dataset for building machine learning models. Furthermore, [PERSON] et al. [22] have designed language qualification tests suitable for annotating tweets. For more complex tasks such as software design, [PERSON] et al. [23] have employed qualification tests to examine the crowd's knowledge and understanding in software user interface design. Moreover, some studies have utilized qualification tests together with other sophisticated approaches to assess the crowd workers. [PERSON] et al. [24] have designed an approach to select reliable crowd workers where they have considered their qualification and behavior while performing the task in which one of the stages for measuring qualification is utilizing qualification tests.  
An in-task screening method includes gold standard questions within the original workflow of the task to estimate the accuracy and reliability of a crowd worker. These questions usually have predefined answers. Crowd workers achieving an accuracy less than a certain threshold are banned from working on the task [14]. Numerous studies have utilized gold standard tests and have set a certain threshold to determine the trustworthiness and reliability of crowd workers.