# Selection of Crowd in Crowdsourcing for Smart Intelligent Applications: A Systematic Mapping Study

Authors: Rui Shang; YongMei Ma; Farhad Ali; ChuanShuang Hu; Shah Nazir; Huafei Wei; Abdullah Khan
Journal: Scientific Programming
 (2021)

---

## Abstract

Crowdsourcing is a task-solving model in which human crowd is hired to solve a particular task. During the crowdsourcing process, the crowd selection is performed in order to select appropriate crowd workers for a specific task; without appropriate selection of crowd workers, the process of crowdsourcing is aimless. The main goal of this paper was to identify the features of crowd in crowdsourcing activity, reasons behind crowd participation in the activity of crowdsourcing, and the existing techniques that were utilized for crowd selection in crowdsourcing. Search strings with corresponding keywords were used to capture relevant studies related to crowdsourcing, and crowd selection was classified under conference papers, journal articles, proceedings, and book chapters. 81 relevant studies are selected from 7 digital data repositories using a search strategy. In crowdsourcing practices, crowd selection was considerably addressed. Nonetheless, it has been noticed that the selection is based only on a single crowd worker attribute such as confidence, past success, efficiency, and experience. For the efficiency and effectiveness of the crowdsourcing operation, crowd selection on multicriteria features is essential.


---

## Body

## 1. Introduction

Crowdsourcing is a task-solving model in which human crowd is employed for solving complex tasks [1]. It is an online activity that is used for gathering the collective knowledge, skills of people in order to complete an ordinary task [2]. Using crowdsourcing, an organization outsources different types of tasks to a huge crowd using an open call. The term crowdsourcing was first coined by Jeff Howe; in the magazine, Wired, he defined it as “the act of taking a job traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call.” Crowdsourcing is a popular approach that is used for performing various activities. The process of crowdsourcing consists of different entities and their interactions such as requester/crowdsourcer/clients who plan, organize, and manage crowdsourcing projects and are capable of submitting the task request, managing the crowd which consists of an online workforce that participates in crowdsourcing tasks or projects, and providing a platform that provides a communication link between the crowd and the requester, i.e., to cope with the task and the crowd [3–7]. Different tasks are performed with the help of crowdsourcing, such as spell checking, creating contents, simple coding, machine learning, distributed solution of a problem like in the development of software and its testing. The tasks are posted on a platform and the crowd participates for performing various types of tasks [6, 8–10]. In crowdsourcing, participants with various backgrounds, levels of qualification, different expertise in various areas, and having various skills and experience work together to perform a specific task or to solve diverse problems [11–14]. Organizations use crowdsourcing applications as the participation of a huge crowd will enable large challenging tasks to be performed or solved in parallel by the crowd. Crowdsourcing is as an effective means for finding a solution toward a corporate problem [15–17]. Crowdsourcing allows organizations (clients) to recruit global, cheap, and skilled work force through online platforms [18, 19].Nowadays, with the help of the Internet, companies are capable of hiring a large group of people with a little effort, as the Internet provides a communication link through which people and organizations can collaborate with each other using different devices such as mobile phones, tablets, iPad, and wearable devices [1, 9, 20–23], and platforms like Amazon Mechanical Turk (AMT) are used by organizations to hire a huge crowd from the global community for accomplishing different types of tasks [24]. In crowdsourcing, an organization can obtain high-quality solutions in less time with a little investment, as it allows a lot of people to complete various tasks with negligible employment and management expenses [25, 26]. As crowdsourcing is an Internet-enabled activity, the online platforms give rise to certain challenges such as the issue of how to announce tasks or jobs, how to select an appropriate platform for submitting task requests, and how to recruit competent people [27]. Recruitment of workers becomes a challenging phase with the increase in the popularity of crowdsourcing. In a large crowd of humans, there might be an untrustworthy participant that often makes mistakes in solving different types of tasks. The identification of the right worker makes the crowdsourcing activity successful [14, 28, 29] for the enhancement of developmental process units, if right skill crowd is hired [30].In a crowdsourcing task, workers’ behaviors and their attitudes are significant features for engaging workers. The crowd must perform the task in the best possible way as the engagement of the crowd depends highly on valuable outcomes. The engagement process will make recruiters able to engage the right worker from a large pool of the crowd. Worker engagement might be influenced by work and worker characteristics. Work characteristic includes task characteristics in which the engagement of the crowd depends highly on the autonomy of task, and it will increase the level of participation with the flexibility and mobility in tasks. The task dissimilarity may influence the participation of workers as irresponsible workers might violate rules of a task. The performance of crowd workers will be increased with the feedback related to their task and they will carefully perform various tasks. Crowd worker’s visibility is the second worker characteristic that has a positive impact on the engagement of the crowd as participants will be visible to organizations. By an increase in the visibility level, workers might be identified among a large crowd pool. The third characteristic of work is Work Setting, on engaging workers for performing a task the worker setting and environmental characteristics play an important role. Organizations have provided a number of options for workers to participate. Crowds can participate virtually from homes, shops, and even from other places of employment. With the increase in potential work setting, the availability, flexibility, and independence of workers is increased, and it will probably increase the performance and satisfaction of workers. The fourth work characteristic is worker recognition, which has practical influence on participant engagement. There are certain factors that drive worker engagement in a job. The unknown nature of crowd work brings some of the challenges that are concerned with the best identification and recognition of crowd workers and how to reward high performers. Personal recognition might be affected by anonymity in crowd settings. Recognition and rewarding participants has shown that the achievements directly impact attitudes and behaviors of workers. Recognition may increase worker engagement by revealing to the crowd that they are valuable assets of the organization. Individuals enroll for complex tasks not because of reward but they think that they will be valued by the organization. By receiving recognition from the organization, individual involvement will be improved and one will put in his or her best efforts and concentration in performing tasks.Crowd worker characteristics are also responsible for worker engagement. It includes Worker Expectations, Extrinsic Motivation, Satisfying Motivation, and Meaningfulness. Worker expectations represent the expectation of crowd workers in an activity from client organizations for their participation in performing a task. The workers who register themselves for performing tasks have certain expectations regarding requirement of the task, period of completion, complexity, and cognitive load that is mandatory for completing the task. When these expectations are violated, they impact comfort, motivation, and satisfaction of workers. Before participation, workers are provided the relevant information about the task such as requirement of the task, the period of completion, and the exact remuneration to participants. With this, the crowd can participate and select the tasks in which they are interested and which matches their skills level. A worker will participate in a task if the task is simple, only less effort is required for thinking, and can be completed in less time. If workers are not provided with task information, they might participate in complex tasks, might become frustrated with the increase in mental effort, and will waste much time. This will affect the behavior and attitudes of workers, thereby lowering the engagement of workers. Crowd workers may be engaged due to the extrinsic motivation factor in which the crowd participates in activities and they are provided with money or reward. There is a liner relation between crowd engagement and extrinsic motivation. Another characteristic is Satisfying Motivation in which the crowd workers expend their cognition, physical, and emotional energies for performing various tasks. Individuals contribute with a goal for achieving the best probable outcomes (example, task performance level). For attaining this, participants use their efforts and concentrate on tasks, and perform well beyond the requirements of satisfiers. A crowd worker depends on resources for achieving high performance level and this will increase the level of engagement of workers. Contrary to satisfying motivation, participants consume their energy, which is required to generate a “good enough” result, and, as a result, decrease the complete engagement of participants in a task. Meaningfulness is another crowd worker characteristic for engaging a crowd; it reflects the degree to which the participants analyze that their contribution is important. Their work is appreciated, valued by clients. This can impact working outcomes such as performance, motivation, and satisfaction. By observing that the tasks performed by workers are valued and accepted, i.e., their efforts are not countless, their interest in engagement will be increased and they will spend more energy in the accomplishment of tasks [31].Analyzing crowdsourcing and its importance in today’s business world, a systematic literature review was carried out for the last ten years, i.e., 2010–2020. The proposed research study is composed of three-fold contributions that are enumerated as follows:(i)
To identify the significant features of the crowd(ii)
To pinpoint the reasons behind the crowd for active participation in crowdsourcing activities(iii)
To explore various existing methods utilized for crowd selection in crowdsourcingThe presented systematic literature review (SLR) mainly focuses on crowd selection that will be used by various organizations to capture the appropriate crowd from a global community to perform various tasks.The paper is structured in a linear fashion and is categorized into 9 sections. Section2 briefly describes and discusses the overall research process carried out for this review. Section 3 explains the validation and threats for this review. Section 4 presents results and discussions of analyzed papers. Section 5 presents the research findings of the review. Section 6 briefly discusses the review topic. Section 7 concludes the review. Section 8 presents the limitation of the proposed review. Section 9 presents future research suggestions.


## 2. Research Methodology

With the interest to analyze the feature, motives of the crowd, and strategies used for the selection of a crowd in a crowdsourcing activity, a systematic literature review was carried out. SLR is a systematic practice for identifying and analyzing related studies as a regard to a specific interest area [32]. For ensuring validity and the value of SLR, a number of standards and protocols are applied by researchers. We adopted the protocol suggested by [33] in carrying out this research. The review protocol and its stages are represented in Figure 1. The review protocol comprised seven stages, namely, analysis of research topic, extraction of research questions, designing a search strategy, extraction of results from data, scrutiny, criteria for quality assessment, and synthesis of data. Research questions are extracted from various studies in the second stage; in the third stage, the searching strategies are designed in accordance with research questions that consist of search term identification and selection of literature resources; collection of extracted data takes place in the fourth stage of the protocol; the fifth stage is concerned with scrutinizing, i.e., refinement of extracted data; scrutinized studies are then evaluated based on a criteria of quality assessment in the six stage. In stage seven, selections of final prospective studies for analysis are carried out.Figure 1
Stages of review protocol.The following sections show the details of the research method.


### 2.1. Research Topic Analysis

Research papers from various digital data repositories were thoroughly reviewed to know the crowdsourcing concept, to discover the issue related to the field, and to find out what researchers have done so far to solve many problems over time.


### 2.2. Research Question

For getting a better perspective on crowdsourcing conception and to identify areas for researchers, a thorough assessment was carried out on the topic of crowdsourcing. Relevant papers, conference proceedings, book chapters, and journals were studied at the initial stage that clearly defined crowdsourcing activity. Our preliminary study identified that crowd selection is a problem-solving technique that is utilized by many organizations for the accomplishment of various tasks by harnessing the collective energy of humans. In order to carry out the research, the proposed study focuses on addressing some research questions given in Table1 that are extracted based on the assessment of various articles/papers for the purpose of making this review an effective one.Table 1
Representation of the research question.
#Research questionsAims1What are the significant features of the crowd for effective crowdsourcing?In the crowdsourcing environment, the participating crowd has some features that are significant for assigning a task. We will analyze many of these features to make crowdsourcing activity efficient and effective.2What are the reasons behind the crowd to actively participate in crowdsourcing?Nobody does anything free; everyone wants something in return. We have to point out the reason as to why the crowd participates in crowdsourcing3What are the existing techniques used for crowd selection in crowdsourcing?Various techniques were used in past for crowd selection as without selecting the appropriate worker, the crowdsourcing activity is aimless. Existing techniques will be discussed briefly.


### 2.3. Searching Strategy

Search terms, literature resources, and search processes make up the search strategy. The descriptions of each are discussed in subsequent steps.


#### 2.3.1. Search String

For creating search terms, the following steps are carried out:(i)
From formulated research questions, major terms were derived(ii)
Synonyms are identified for main terms(iii)
Keywords were identified from relevant books or papers(iv)
Boolean OR is used to link synonyms(v)
Main terms are linked with Boolean ANDThe strings that are formed as a result are presented as follows:(i)
(Multicriteria OR crowd OR crowd selection OR worker selection) AND (crowdsourcing) AND (software OR software engineering)(ii)
(“Multicriteria” OR “crowd selection” OR “worker selection”) AND (crowdsourcing) AND (software engineering)


#### 2.3.2. Literature Resources

For synchronization, a thorough assessment of the proposed study was carried out on seven digital libraries to extract data from various scholars’ works on the topic of crowdsourcing. These digital libraries include: ScienceDirect, IEEE Xplore, ACM Digital Library, Springer, Taylor & Francis, Wiley, and Hindawi. For carrying out this research title, abstracts, index terms, contents of conference proceedings, published research papers, journal articles, book chapters, and conference papers were analyzed.


#### 2.3.3. The Search Process

For a comprehensive search, a systematic literature review of relevant sources is significant to be carried out. Yet, the search process that is utilized in the proposed study consists of some stages that are mentioned below. These steps are also represented in Figure2:(i)
Stage 1: seven digital libraries are thoroughly searched to extract papers related to the proposed research. The search result has been assembled to be a collection of prospective papers.(ii)
Stage 2: on the basis of titles, papers were filtered from total papers.(iii)
Stage 3: relevant papers were extracted by applying quality assessment criteria.Figure 2
Representation of different stages of our search process.


### 2.4. Prospective Study Selection

The first stage of searching retrieved 2098 metadata studies (abstract, title, contents) from seven digital databases. After this stage, the titles of perspective studies were used for extracting relevant research work. This process is mandatory for the exclusion of irrelevant and duplicate research studies. After applying title base filtering, 146 related studies were identified. Finally, we applied quality assessment criteria (QoA) on the filtered papers, i.e., each paper was studied and assessed in detail to check whether these studies answer the questions that were formulated. At the end of this process, 81 relevant studies capable of answering the questions were selected.


#### 2.4.1. Scrutinization Process

Scrutiny was necessary to extract the relevant work as we have obtained 2098 prospective studies in the initial search process. At first, title-wise paper selection was used to filter relevant papers. Next, the relevant studies are analyzed on the basis of paper contents. Hence, all articles that were unable to address our research question and do not reveal the discussion topic were excluded. Studies that are published in conferences, proceedings, journal publications, and written only in English were included in the relevant studies. Duplicate research studies were also eliminated in this process. A systematic literature review was carried out for a period of ten years, i.e., 2010–2020, in order to select recent studies on the topic. Our initial search process on seven digital databases was conducted between January 24, 2020 and January 29, 2020. The criteria adopted for the process of scrutiny is presented in Table2.Table 2
Criteria for inclusion/exclusion.
Inclusion CriteriaExclusion CriteriaArticles that are published in English languageArticles that are published in other languagesPapers that focus on crowdsourcingPapers that do not discuss the research topicPapers that are published between 2010 and 2020Papers that are published before the year 2010.Papers that are capable of answering minimal two formulated questionsPapers that are incapable of answering two questionsFigure3 shows the type of papers along with the year of publication.Figure 3
Type of papers with the year of publication.Table3 shows the library-wise search based on title and contents.Table 3
Library-wise search based on title and contents.
LibrariesTitle-wise selectionContent-wise selectionHindawi32Wiley33ACM digital library96Springer106Taylor and Francis148IEEE4323Science direct5733Total781Figure4 shows the reference of paper, types of paper with the year of publication.Figure 4
Reference, year, and type of paper.


### 2.5. Quality Assessment (QoA) of the Selected Papers

The quality of assessment was obtained for selected studies by scoring or evaluating these studies in order to check whether these studies are capable of answering formulated research questions or not. Such questions are dealt with in Table4. There are only 2 answers (options) to each question, i.e., “yes” = 1 and “No” = 0. The quality score of each study is calculated as the sum of answering the formulated questions. The reliability of the proposed study was obtained by considering studies that are relevant and whose quality score is two or three, i.e., papers that are capable of answering at least two research questions. Based on quality scoring, 65 studies were excluded and 81 papers were selected as they were capable of answering at least two formulated questions. The quality scores of selected studies are shown in Table 5 and the graphical representation is represented in Figure 5.Table 4
Representation of quality assessment criteria.
S. no.Questions1Are the studies representing the features of the crowd?2Do the papers answer as to why the crowd participates in an activity?3Does the research point out the techniques for crowd selection?Table 5
Quality score results of selected studies.
Cite no.TitleQ1Q2Q3Score[34]A hybrid simulation model for crowdsourced software development1012[35]A vision of crowd development1102[36]Affinity-aware online selection mechanisms in mobile crowdsourcing sensing1113[37]An exploratory study on perception of Indian crowd toward crowdsourcing software development1102[16]Analyzing crowd labor and designing incentives for humans in the loop1102[38]Barriers faced by newcomers to software crowdsourcing projects1102[39]Competence, collaboration, and time management: Barriers and recommendations for crowd workers1113[19]Competition-based crowdsourcing software development: A multi-method study from a customer perspective1102[15]Crowd development1102[40]CrowdService: Serving the individuals through mobile crowdsourcing and service composition0112[41]Crowdsourced software development: Exploring the motivational and inhibiting factors of the South African crowd1102[42]Efficient worker selection through history-based learning in crowdsourcing1113[24]Estimating software task effort in crowds1012[43]Guiding the crowds for android testing1102[44]Leveraging crowdsourcing for team elasticity: An empirical evaluation at TopCoder1012[45]Leveraging the power of the crowd for software testing0112[22]Supporting coordination in crowdsourced software testing services1113[13]Task allocation for crowdsourcing using AI planning1113[4]Toward microtask crowdsourcing software design work0112[28]Toward adopting alternative workforce for software engineering1012[17]Virtual team performance in crowdsourcing contest: A social network perspective1102[46]Who should be selected to perform a task in crowdsourced testing?1012[47]Worker ranking determination in crowdsourcing platforms using aggregation functions1113[48]A fuzzy expert system to trust-based access control in crowdsourcing environments1113[49]A survey of the use of crowdsourcing in software engineering1113[3]A systematic literature review on crowdsourcing in software engineering1113[50]An incentive mechanism with privacy protection in mobile crowdsourcing systems1113[51]A system for scalable and reliable technical-skill testing in online labor markets1012[52]Auction-based crowdsourcing supporting skill management1113[18]Best of both worlds: Mitigating imbalance of crowd worker strategic choices without a budget1102[31]Catering to the crowd: An HRM perspective on crowd worker engagement0112[53]CloudTeams: Bridging the gap between developers and customers during software development processes1113[54]Competition matters! self-efficacy, effort, and performance in crowdsourcing teams1102[12]Cooperation or competition—When do people contribute more? A field experiment on gamification of crowdsourcing1102[11]Crowdsourcing: A taxonomy and systematic mapping study1102[55]Crowdsourcing contests1102[21]Crowdsourcing not all sourced by the crowd: An observation on the behavior of Wikipedia participants1102[56]Efficient crowdsourcing of unknown experts using bounded multiarmed bandits1113[57]Hybrid crowd-based decision support in business processes1113[58]Improving accuracy and lowering cost in crowdsourcing through an unsupervised expertise estimation approach1012[59]Incentivizing social media users for mobile crowdsourcing1113[60]Information technology (IT)-enabled crowdsourcing: A conceptual framework1113[61]Inspiring crowdsourcing communities to create novel solutions: Competition design and the mediating role of trust1102[62]Mobile crowd sensing—Taxonomy, applications, challenges, and solutions0112[63]Modeling cognitive bias in crowdsourcing systems1102[64]Open or proprietary? Choosing the right crowdsourcing platform for innovation1102[23]Privacy-preserving QoI-aware participant coordination for mobile crowdsourcing0112[2]Real-time crowdsourcing with payment of idle workers in the retainer model1102[65]SenseChain: A blockchain-based crowdsensing framework for multiple requesters and multiple workers0112[14]The wisdom of crowds: The potential of online communities as a tool for data analysis1102[30]Toward collaborative software engineering leveraging the crowd1102[8]Trait motivations of crowdsourcing and task choice: A distal-proximal perspective1102[9]Trust-based privacy-aware participant selection in social participatory sensing0112[6]Understanding crowdsourcing projects: A systematic review of tendencies, workflow, and quality management1113[66]Weaving risk identification into crowdsourcing lifecycle0112[67]Winners, losers, and deniers: Self-selection in crowd innovation contests and the roles of motivation, creativity, and skills1102[29]A context-aware approach for trustworthy worker selection in a social crowd1113[68]A survey of task-oriented crowdsourcing1113[27]An investigation of factors affecting the visits of online crowdsourcing and labor platforms1102[69]Improving reliability of crowdsourced results by detecting crowd workers with multiple identities1102[10]Volunteer selection based on crowdsourcing approach1113[70]Why incorporating a platform-intermediary can increase crowdsourcees’ engagement1102[71]A conceptual framework for increasing innovation through improved selection of specialized professionals1113[72]Facilitating collocated crowdsourcing on situated displays1102[73]Framework and literature analysis for crowdsourcing’s answer aggregation1102[74]Practical POMDP-based test mechanism for quality assurance in volunteer crowdsourcing1102[75]Rules of crowdsourcing: Models, issues, and systems of control1102[76]Skills and wills: The keys to identify the right team in collaborative innovation platforms1012[77]Task design, motivation, and participation in crowdsourcing contests1113[5]Toward an understanding of participants’ sustained participation in crowdsourcing contests1102[20]Crowdsourcing: A review and suggestions for future research1102[7]Of crowds and talents: Discursive constructions of global online labor1113[78]The ethical use of crowdsourcing1102[79]A transfer learning-based framework of crowd-selection on twitter1012[25]Crowd build: A methodology for enterprise software development using crowdsourcing1113[80]CrowdEval: A cost-efficient strategy to evaluate crowdsourced worker’s reliability1113[26]CrowdSelect: Increasing accuracy of crowdsourcing tasks through behavior prediction and user selection1012[81]December: A declarative tool for crowd member selection1012[82]Declarative user selection with soft constraints1012[1]Optimal task partition with delay requirement in mobile crowdsourcing1113[83]TDSRC: A task-distributing system of crowdsourcing based on social relation cognition1113Figure 5
Quality of assessment chart of the selected papers.


### 2.6. Synthesizing Data

The significance of this step is to synthesize and summarize the proofs in order to answer formulated questions. Eighty-one relevant studies have been synthesized using thematic analysis, which is a powerful technique used to rigorously generate themes and patterns for data [84]. A theme extract is something significant about the RQ data, and it can be derived inductively or deductively from the data based on themes derived in previous studies. The process will help in synchronization of the relevant study in order to intensify clarity. This will also assist in identification of specific answers to formulated questions. A summary of data synthesis for the research question is explained below. The data associated with all RQs is represented in a table form and it is also discussed in detail. In RQ1, the features of the crowd are extracted from selected studies. RQ2 explains in detail the reasons for crowd participation in an activity. RQ3 focuses on various techniques used for crowd selection.


## 2.1. Research Topic Analysis

Research papers from various digital data repositories were thoroughly reviewed to know the crowdsourcing concept, to discover the issue related to the field, and to find out what researchers have done so far to solve many problems over time.


## 2.2. Research Question

For getting a better perspective on crowdsourcing conception and to identify areas for researchers, a thorough assessment was carried out on the topic of crowdsourcing. Relevant papers, conference proceedings, book chapters, and journals were studied at the initial stage that clearly defined crowdsourcing activity. Our preliminary study identified that crowd selection is a problem-solving technique that is utilized by many organizations for the accomplishment of various tasks by harnessing the collective energy of humans. In order to carry out the research, the proposed study focuses on addressing some research questions given in Table1 that are extracted based on the assessment of various articles/papers for the purpose of making this review an effective one.Table 1
Representation of the research question.
#Research questionsAims1What are the significant features of the crowd for effective crowdsourcing?In the crowdsourcing environment, the participating crowd has some features that are significant for assigning a task. We will analyze many of these features to make crowdsourcing activity efficient and effective.2What are the reasons behind the crowd to actively participate in crowdsourcing?Nobody does anything free; everyone wants something in return. We have to point out the reason as to why the crowd participates in crowdsourcing3What are the existing techniques used for crowd selection in crowdsourcing?Various techniques were used in past for crowd selection as without selecting the appropriate worker, the crowdsourcing activity is aimless. Existing techniques will be discussed briefly.


## 2.3. Searching Strategy

Search terms, literature resources, and search processes make up the search strategy. The descriptions of each are discussed in subsequent steps.


### 2.3.1. Search String

For creating search terms, the following steps are carried out:(i)
From formulated research questions, major terms were derived(ii)
Synonyms are identified for main terms(iii)
Keywords were identified from relevant books or papers(iv)
Boolean OR is used to link synonyms(v)
Main terms are linked with Boolean ANDThe strings that are formed as a result are presented as follows:(i)
(Multicriteria OR crowd OR crowd selection OR worker selection) AND (crowdsourcing) AND (software OR software engineering)(ii)
(“Multicriteria” OR “crowd selection” OR “worker selection”) AND (crowdsourcing) AND (software engineering)


### 2.3.2. Literature Resources

For synchronization, a thorough assessment of the proposed study was carried out on seven digital libraries to extract data from various scholars’ works on the topic of crowdsourcing. These digital libraries include: ScienceDirect, IEEE Xplore, ACM Digital Library, Springer, Taylor & Francis, Wiley, and Hindawi. For carrying out this research title, abstracts, index terms, contents of conference proceedings, published research papers, journal articles, book chapters, and conference papers were analyzed.


### 2.3.3. The Search Process

For a comprehensive search, a systematic literature review of relevant sources is significant to be carried out. Yet, the search process that is utilized in the proposed study consists of some stages that are mentioned below. These steps are also represented in Figure2:(i)
Stage 1: seven digital libraries are thoroughly searched to extract papers related to the proposed research. The search result has been assembled to be a collection of prospective papers.(ii)
Stage 2: on the basis of titles, papers were filtered from total papers.(iii)
Stage 3: relevant papers were extracted by applying quality assessment criteria.Figure 2
Representation of different stages of our search process.


## 2.3.1. Search String

For creating search terms, the following steps are carried out:(i)
From formulated research questions, major terms were derived(ii)
Synonyms are identified for main terms(iii)
Keywords were identified from relevant books or papers(iv)
Boolean OR is used to link synonyms(v)
Main terms are linked with Boolean ANDThe strings that are formed as a result are presented as follows:(i)
(Multicriteria OR crowd OR crowd selection OR worker selection) AND (crowdsourcing) AND (software OR software engineering)(ii)
(“Multicriteria” OR “crowd selection” OR “worker selection”) AND (crowdsourcing) AND (software engineering)


## 2.3.2. Literature Resources

For synchronization, a thorough assessment of the proposed study was carried out on seven digital libraries to extract data from various scholars’ works on the topic of crowdsourcing. These digital libraries include: ScienceDirect, IEEE Xplore, ACM Digital Library, Springer, Taylor & Francis, Wiley, and Hindawi. For carrying out this research title, abstracts, index terms, contents of conference proceedings, published research papers, journal articles, book chapters, and conference papers were analyzed.


## 2.3.3. The Search Process

For a comprehensive search, a systematic literature review of relevant sources is significant to be carried out. Yet, the search process that is utilized in the proposed study consists of some stages that are mentioned below. These steps are also represented in Figure2:(i)
Stage 1: seven digital libraries are thoroughly searched to extract papers related to the proposed research. The search result has been assembled to be a collection of prospective papers.(ii)
Stage 2: on the basis of titles, papers were filtered from total papers.(iii)
Stage 3: relevant papers were extracted by applying quality assessment criteria.Figure 2
Representation of different stages of our search process.


## 2.4. Prospective Study Selection

The first stage of searching retrieved 2098 metadata studies (abstract, title, contents) from seven digital databases. After this stage, the titles of perspective studies were used for extracting relevant research work. This process is mandatory for the exclusion of irrelevant and duplicate research studies. After applying title base filtering, 146 related studies were identified. Finally, we applied quality assessment criteria (QoA) on the filtered papers, i.e., each paper was studied and assessed in detail to check whether these studies answer the questions that were formulated. At the end of this process, 81 relevant studies capable of answering the questions were selected.


### 2.4.1. Scrutinization Process

Scrutiny was necessary to extract the relevant work as we have obtained 2098 prospective studies in the initial search process. At first, title-wise paper selection was used to filter relevant papers. Next, the relevant studies are analyzed on the basis of paper contents. Hence, all articles that were unable to address our research question and do not reveal the discussion topic were excluded. Studies that are published in conferences, proceedings, journal publications, and written only in English were included in the relevant studies. Duplicate research studies were also eliminated in this process. A systematic literature review was carried out for a period of ten years, i.e., 2010–2020, in order to select recent studies on the topic. Our initial search process on seven digital databases was conducted between January 24, 2020 and January 29, 2020. The criteria adopted for the process of scrutiny is presented in Table2.Table 2
Criteria for inclusion/exclusion.
Inclusion CriteriaExclusion CriteriaArticles that are published in English languageArticles that are published in other languagesPapers that focus on crowdsourcingPapers that do not discuss the research topicPapers that are published between 2010 and 2020Papers that are published before the year 2010.Papers that are capable of answering minimal two formulated questionsPapers that are incapable of answering two questionsFigure3 shows the type of papers along with the year of publication.Figure 3
Type of papers with the year of publication.Table3 shows the library-wise search based on title and contents.Table 3
Library-wise search based on title and contents.
LibrariesTitle-wise selectionContent-wise selectionHindawi32Wiley33ACM digital library96Springer106Taylor and Francis148IEEE4323Science direct5733Total781Figure4 shows the reference of paper, types of paper with the year of publication.Figure 4
Reference, year, and type of paper.


## 2.4.1. Scrutinization Process

Scrutiny was necessary to extract the relevant work as we have obtained 2098 prospective studies in the initial search process. At first, title-wise paper selection was used to filter relevant papers. Next, the relevant studies are analyzed on the basis of paper contents. Hence, all articles that were unable to address our research question and do not reveal the discussion topic were excluded. Studies that are published in conferences, proceedings, journal publications, and written only in English were included in the relevant studies. Duplicate research studies were also eliminated in this process. A systematic literature review was carried out for a period of ten years, i.e., 2010–2020, in order to select recent studies on the topic. Our initial search process on seven digital databases was conducted between January 24, 2020 and January 29, 2020. The criteria adopted for the process of scrutiny is presented in Table2.Table 2
Criteria for inclusion/exclusion.
Inclusion CriteriaExclusion CriteriaArticles that are published in English languageArticles that are published in other languagesPapers that focus on crowdsourcingPapers that do not discuss the research topicPapers that are published between 2010 and 2020Papers that are published before the year 2010.Papers that are capable of answering minimal two formulated questionsPapers that are incapable of answering two questionsFigure3 shows the type of papers along with the year of publication.Figure 3
Type of papers with the year of publication.Table3 shows the library-wise search based on title and contents.Table 3
Library-wise search based on title and contents.
LibrariesTitle-wise selectionContent-wise selectionHindawi32Wiley33ACM digital library96Springer106Taylor and Francis148IEEE4323Science direct5733Total781Figure4 shows the reference of paper, types of paper with the year of publication.Figure 4
Reference, year, and type of paper.


## 2.5. Quality Assessment (QoA) of the Selected Papers

The quality of assessment was obtained for selected studies by scoring or evaluating these studies in order to check whether these studies are capable of answering formulated research questions or not. Such questions are dealt with in Table4. There are only 2 answers (options) to each question, i.e., “yes” = 1 and “No” = 0. The quality score of each study is calculated as the sum of answering the formulated questions. The reliability of the proposed study was obtained by considering studies that are relevant and whose quality score is two or three, i.e., papers that are capable of answering at least two research questions. Based on quality scoring, 65 studies were excluded and 81 papers were selected as they were capable of answering at least two formulated questions. The quality scores of selected studies are shown in Table 5 and the graphical representation is represented in Figure 5.Table 4
Representation of quality assessment criteria.
S. no.Questions1Are the studies representing the features of the crowd?2Do the papers answer as to why the crowd participates in an activity?3Does the research point out the techniques for crowd selection?Table 5
Quality score results of selected studies.
Cite no.TitleQ1Q2Q3Score[34]A hybrid simulation model for crowdsourced software development1012[35]A vision of crowd development1102[36]Affinity-aware online selection mechanisms in mobile crowdsourcing sensing1113[37]An exploratory study on perception of Indian crowd toward crowdsourcing software development1102[16]Analyzing crowd labor and designing incentives for humans in the loop1102[38]Barriers faced by newcomers to software crowdsourcing projects1102[39]Competence, collaboration, and time management: Barriers and recommendations for crowd workers1113[19]Competition-based crowdsourcing software development: A multi-method study from a customer perspective1102[15]Crowd development1102[40]CrowdService: Serving the individuals through mobile crowdsourcing and service composition0112[41]Crowdsourced software development: Exploring the motivational and inhibiting factors of the South African crowd1102[42]Efficient worker selection through history-based learning in crowdsourcing1113[24]Estimating software task effort in crowds1012[43]Guiding the crowds for android testing1102[44]Leveraging crowdsourcing for team elasticity: An empirical evaluation at TopCoder1012[45]Leveraging the power of the crowd for software testing0112[22]Supporting coordination in crowdsourced software testing services1113[13]Task allocation for crowdsourcing using AI planning1113[4]Toward microtask crowdsourcing software design work0112[28]Toward adopting alternative workforce for software engineering1012[17]Virtual team performance in crowdsourcing contest: A social network perspective1102[46]Who should be selected to perform a task in crowdsourced testing?1012[47]Worker ranking determination in crowdsourcing platforms using aggregation functions1113[48]A fuzzy expert system to trust-based access control in crowdsourcing environments1113[49]A survey of the use of crowdsourcing in software engineering1113[3]A systematic literature review on crowdsourcing in software engineering1113[50]An incentive mechanism with privacy protection in mobile crowdsourcing systems1113[51]A system for scalable and reliable technical-skill testing in online labor markets1012[52]Auction-based crowdsourcing supporting skill management1113[18]Best of both worlds: Mitigating imbalance of crowd worker strategic choices without a budget1102[31]Catering to the crowd: An HRM perspective on crowd worker engagement0112[53]CloudTeams: Bridging the gap between developers and customers during software development processes1113[54]Competition matters! self-efficacy, effort, and performance in crowdsourcing teams1102[12]Cooperation or competition—When do people contribute more? A field experiment on gamification of crowdsourcing1102[11]Crowdsourcing: A taxonomy and systematic mapping study1102[55]Crowdsourcing contests1102[21]Crowdsourcing not all sourced by the crowd: An observation on the behavior of Wikipedia participants1102[56]Efficient crowdsourcing of unknown experts using bounded multiarmed bandits1113[57]Hybrid crowd-based decision support in business processes1113[58]Improving accuracy and lowering cost in crowdsourcing through an unsupervised expertise estimation approach1012[59]Incentivizing social media users for mobile crowdsourcing1113[60]Information technology (IT)-enabled crowdsourcing: A conceptual framework1113[61]Inspiring crowdsourcing communities to create novel solutions: Competition design and the mediating role of trust1102[62]Mobile crowd sensing—Taxonomy, applications, challenges, and solutions0112[63]Modeling cognitive bias in crowdsourcing systems1102[64]Open or proprietary? Choosing the right crowdsourcing platform for innovation1102[23]Privacy-preserving QoI-aware participant coordination for mobile crowdsourcing0112[2]Real-time crowdsourcing with payment of idle workers in the retainer model1102[65]SenseChain: A blockchain-based crowdsensing framework for multiple requesters and multiple workers0112[14]The wisdom of crowds: The potential of online communities as a tool for data analysis1102[30]Toward collaborative software engineering leveraging the crowd1102[8]Trait motivations of crowdsourcing and task choice: A distal-proximal perspective1102[9]Trust-based privacy-aware participant selection in social participatory sensing0112[6]Understanding crowdsourcing projects: A systematic review of tendencies, workflow, and quality management1113[66]Weaving risk identification into crowdsourcing lifecycle0112[67]Winners, losers, and deniers: Self-selection in crowd innovation contests and the roles of motivation, creativity, and skills1102[29]A context-aware approach for trustworthy worker selection in a social crowd1113[68]A survey of task-oriented crowdsourcing1113[27]An investigation of factors affecting the visits of online crowdsourcing and labor platforms1102[69]Improving reliability of crowdsourced results by detecting crowd workers with multiple identities1102[10]Volunteer selection based on crowdsourcing approach1113[70]Why incorporating a platform-intermediary can increase crowdsourcees’ engagement1102[71]A conceptual framework for increasing innovation through improved selection of specialized professionals1113[72]Facilitating collocated crowdsourcing on situated displays1102[73]Framework and literature analysis for crowdsourcing’s answer aggregation1102[74]Practical POMDP-based test mechanism for quality assurance in volunteer crowdsourcing1102[75]Rules of crowdsourcing: Models, issues, and systems of control1102[76]Skills and wills: The keys to identify the right team in collaborative innovation platforms1012[77]Task design, motivation, and participation in crowdsourcing contests1113[5]Toward an understanding of participants’ sustained participation in crowdsourcing contests1102[20]Crowdsourcing: A review and suggestions for future research1102[7]Of crowds and talents: Discursive constructions of global online labor1113[78]The ethical use of crowdsourcing1102[79]A transfer learning-based framework of crowd-selection on twitter1012[25]Crowd build: A methodology for enterprise software development using crowdsourcing1113[80]CrowdEval: A cost-efficient strategy to evaluate crowdsourced worker’s reliability1113[26]CrowdSelect: Increasing accuracy of crowdsourcing tasks through behavior prediction and user selection1012[81]December: A declarative tool for crowd member selection1012[82]Declarative user selection with soft constraints1012[1]Optimal task partition with delay requirement in mobile crowdsourcing1113[83]TDSRC: A task-distributing system of crowdsourcing based on social relation cognition1113Figure 5
Quality of assessment chart of the selected papers.


## 2.6. Synthesizing Data

The significance of this step is to synthesize and summarize the proofs in order to answer formulated questions. Eighty-one relevant studies have been synthesized using thematic analysis, which is a powerful technique used to rigorously generate themes and patterns for data [84]. A theme extract is something significant about the RQ data, and it can be derived inductively or deductively from the data based on themes derived in previous studies. The process will help in synchronization of the relevant study in order to intensify clarity. This will also assist in identification of specific answers to formulated questions. A summary of data synthesis for the research question is explained below. The data associated with all RQs is represented in a table form and it is also discussed in detail. In RQ1, the features of the crowd are extracted from selected studies. RQ2 explains in detail the reasons for crowd participation in an activity. RQ3 focuses on various techniques used for crowd selection.


## 3. Results and Discussion

This segment summarizes the findings of the study. First, we present an overview of selected articles. Next, we present in separate subsections a description of the results of the review process, in accordance with the questions formulated.


### 3.1. Overview of Selected Papers

For the proposed study, 81 Research studies are selected on the basis of QoA criteria. Among these studies, 55 papers were published in journal articles, 1 paper in book section, 20 are extracted from conference proceedings, and 5 studies were retrieved from conference papers. The total number of selected papers and their percentages are presented in Figure6; the selected papers based on publication year are shown in Figure 7.Figure 6
Percentages of selected studies.Figure 7
Number of selected papers based on publication year.


### 3.2. Significant Features of Crowd in Crowdsourcing (RQ1)

Organizations are interested in leveraging and gaining the knowledge of people. Superior techniques are applied for collecting this knowledge from external experts for enhancing the performance of products. Managers of an organization who are interested in solving challenges must select specialized professionals or subject experts from the external world who have the knowledge for solving a new organizational challenge. These specialized persons represent a solid approach to increase the absorptive capacity or the company’s capability to identify, assimilate, and exploit external knowledge. Various approaches are used by specialized professionals for increasing the storage capacity, or they may use the potential of the company for identifying, selecting, assimilating, and utilizing the knowledge of experts [71]. In crowdsourcing, the entire project or a part of it is outsourced to a large pool [39], of distributive and cost effective’s labors [72]. Crowd workers possess some characteristics, i.e., they can be identified with certain rules that direct their behaviors, they do not depend on others, they control their actions, and they may be flexible [34, 81] in changing their behavior with the changing environment [34]. Crowd workers may be newcomers (novices) [12, 20, 41, 47]; anonymous participants [3, 6, 7, 11, 19, 73, 83]; inexperienced [11], fraudulent [1], incompetent [21] workers; heterogeneous workers [38, 52, 56, 64] in working styles; unknown workers [73]; transient solvers (it greatly influences the quality of a task) [83]; undefined workers [3]; nonprofessionals [49, 56, 83]; nonexperts [49, 68]; untrained participants [3, 35]; untrustworthy workers only interested in gaining the rewards that are associated with tasks and thus do not work sincerely, and this is a negative factor of the crowd [29]. They act selfishly (negative feature) for utility utilization [18]. Workers may act maliciously [35, 80, 83],which limits the quality of tasks, or may be “eager beavers” [35] who outperform appropriate workers [5]. Trainees may accomplish tasks [3, 41] in the crowdsourcing activity. The crowd may be autonomous [34], fast [2] unique [69], appropriate [3], right [28, 79], reliable/efficient [2, 25, 72, 73, 80], loyal [70], truthful [36], trustworthy [29, 42, 44, 50, 63] who complete assigned tasks sincerely. Workers are coordinative, adaptable [16] (as they can change themselves with the change in the work environment), “Energetic” having energy [10, 17], capable [1, 42, 60] of performing tasks, and are “creative” as they creatively [67, 68, 78] perform different types of work. They are competent workers [16, 39, 63] who utilize common sense [68] in various tasks. Crowds are smart [25, 27], educated/qualified [1, 13, 42], professionals [71, 75], having skills (skillful workers) [2, 8, 10, 12, 14, 17, 26, 28, 37, 48, 51, 54, 56, 57, 59–61, 67, 76–78], expertise [58, 59, 81, 82], experience [14, 17, 19, 24, 43, 46, 47, 53, 76] who possess knowledge [2, 8, 11, 13–15, 17, 19, 22, 30, 35, 38, 49, 55, 60, 61, 68, 70, 71, 75, 76] for getting a task done [35]. Workers may be cooperative [12], collaborative [21, 39, 76]. Crowds may work as volunteers [10, 15, 17, 74–76] for solving large problems [5, 7, 21, 54, 60, 64, 68, 75, 78], i.e., they are solution providers [5]. Workers can make decisions [15].The various features of the crowd are represented in Table 6 and Table 7.Table 6
Negative features/characteristics of crowd workers.
S. no.Negative featuresCitations1Nonprofessionals[49, 56, 83]2Nonexperts[49, 68]3Untrustworthy[29]4Unskilled workers[48]5Anonymous[3, 6, 7, 11, 19, 73, 83]6Untrained participants[3, 35]7Malicious workers[35, 80, 83]8Selfish workers[18]9Novice[12, 20, 41, 47]10Transient workers[83]11Inexperience[11]12Fraudulent[1]13Incompetence[21]Table 7
Positive features/characteristics of crowd workers.
S. no.Positive featuresCitation1Professionals[71, 75]2Expertise[58, 59, 81, 82]3Trustworthy[29, 42, 44, 50, 63]4Skill[2, 8, 10, 12, 14, 17, 26, 28, 37, 48, 51, 54, 56, 57, 59–61, 67, 76–78]5Identifiable[34]6Autonomous[34]7Flexible[34, 81]8Creative[67, 68, 78]9Common sense[68]10Knowledge[2, 8, 11, 13–15, 17, 19, 22, 30, 35, 38, 49, 55, 60, 61, 68, 70, 71, 75, 76]11Appropriate[3]12Eager beavers[35]13Trainee[3, 41]14Competent[16, 39, 63]15Adaptable[16]16Coordinative[16]17Heterogeneous[38, 52, 56, 64]18Collaborative[21, 39, 76]19Decision maker[15]20Qualified/educated[1, 13, 42]21Ability[1, 42, 60]22Problem-solving[5, 7, 21, 54, 60, 64, 68, 75, 78]23Unique (workers)[69]24Volunteers[10, 15, 17, 74–76]25Energetic[10, 17]26Experienced[14, 17, 19, 24, 43, 46, 47, 53, 76]27Reliable/efficient (workers)[2, 25, 72, 73, 80]28Accurate[74]29Smart[25, 27]30Cost-effective workers[72]31Solution providers[5]32Right[28, 79]33Fast[2]34Cooperative[12]35Truthful[36]36Inventive[64]37Loyal[70]


### 3.3. Reasons behind Crowd Participation (RQ2)

Various incentive measures are adopted to motivate participant for contribution like for teams in organizations, pointing system is adopted and participants are provided with different points depending on their performance level. These points are then converted into prizes and gifts. External participants are provided with honoraria and likelihood of career prospects/advancement [14, 19, 20, 41] and being employed in future work [71]. Various workers complete different types of tasks such as sentiment analysis and tagging of image, solving microprogramming tasks in order to receive benefits of remuneration [2, 19, 41, 56] financial [6], monetary [41, 47, 83], reward [37, 78], prize [36, 37, 49, 71, 78], payment [45, 65, 66] like immediate payoffs (making money) [5], cash [50, 75], fee [57], direct compensation [16, 31, 61, 66] and money [8, 55], extra bonuses [12, 35], extra income [43], non-monetary awards including delayed payoffs (the potential to take freelance work) [5], status [3], pleasure personal enjoyment/fun [5, 62], and for learning purpose [6, 45] to increase their knowledge [15–17], increase reputation [65, 77] in public display by increase in point and badges. The motivation may be an introjected one in which crowd participates for recognition [6, 62] among peers, fame [37]. People also compete with each other to develop their skills [6, 45]. Crowds are rewarded based on the quality of the completed task. A worker who produces quality results is awarded with incentives [35, 37] and access for future work [8, 43]. Worker observes developers [15] work, raise, and collecting funds [12, 49, 53, 75] for projects. Crowd also participates for inspiration [49] from other solutions and revision of solutions for generating better alternative [3, 17] for existing solutions. Communication [39, 53] with each other to manage their time [39] is also a reason for participation. Crowd works in cooperation [15, 20] and in collaboration [12, 53] for improving products. Crowd participants may also participate to provide ideas [12, 53] for various developmental projects. A participant participates as he/she perceives curiosity [12, 60, 77], self-affirmation [60], self-estimation [6, 78], and appreciation [49] of requester organizations. Workers are motivated for self-marketability [41, 77], visibility [7]. Self-efficacy [20, 41, 54] which constitutes four types of incentives, namely, accomplishment [12, 40, 41] (workers feel that they are accomplishing something while participating in tasks), importance (workers observe that their doing is important to others), competence [41] (working on crowdsourcing tasks provide a wisdom of competency), and doing best job (workers desire to know that they are doing best), and to contribute. There are four incentives regarding contribution, namely, effectiveness (how effective the tasks are to others), helping others (participant completes the task to help other people), trust (workers are trusted by organizations), and supporting others (participants support each other in the accomplishment of tasks) [41]. The crowd also participates to find an appropriate answer for a question [3, 35, 49]. An individual can be motivated to gain or share experience [12, 62] and increase his or her expertise level [59]. Other reasons for motivating the crowd to participate in tasks are volunteering (free contribution) [6, 10, 74], reciprocity and expectancy in which users participate for intellectual stimulation [2]. To help a community as they expect that others will also contribute in society [2, 41, 77], contribution of each every one will help in solving a community problem. Crowd contributes altruistically [9] without greediness/expectations for something; they do it out of sympathy, which is known as altruism [62]. There are also some factors that motivate the crowd for participation like making new friends [16] and for socialization [54, 70] purposes; people also contribute in crowdsourcing as they have interest in [16, 60, 65] a field. Developers for task are motivated by ideology, self-need, self-development, and for preserving authorship of their own work to get paid to provide best solutions to problems [39]. Crowds may provide a service [2, 40], acquire information [40, 62], or may accomplish a task like booking a table in a restaurant [40]. Contestants also participate so that organizations provide them feedback [47, 55] on their completed work. The crowds are motivated by means of intrinsic and extrinsic motivation [77] Intrinsic motivation is concerned with participation that increases interest and joy due to which participant feels competency, fulfillment, and satisfaction [12, 61]. Extrinsic motivation is working on something that provides some distinguishable results. It involves engagement because of external incentives such as compensation [66]. The participation may be due to identifiable motivation, which is a reaction of freedom and violation since the deeds are concerned with identity and personal goal achievements [61]. Integrative motives [60] may be a cause of participation in which participant activities are considered significant and meaningful. The crowd is also motivated by networking motives for building professional or personal relation [17, 60]. This will increase self-belonging [60] and increase engagement. Self-marketing [41, 60, 77] or self-advertisement [60, 61] is a major motivation for participation in projects for individuals looking for a job. Workers perceive that competition provides workers with a number of benefits, and they have to show eagerness to engage in competitions [15, 16, 61]. Workers are willing to participate as they are provided with task autonomy [61]; participants are offered a high level of control on their actions in competition. Task variety is another motive for participation where a participant is allowed to provide solutions from different viewpoints. Players are allowed to use their abilities and skills for developing solutions. Task complexity is another motive for participation as task is complex; its completion requires high competency, skills. Workers are interested in performing complex tasks to gain self-expression and a sense of competency. The crowd also participates for sponsoring an organization [61]. The crowd also shares and acquires information [40, 62], recommends [62, 75] some things, shares life experiences, provides suggestion, and increases the level of ranking [62]. Crowd teams participate in crowd source task for receiving tangible (monetary) and intangible (reputation) rewards, for social comparison (self-evaluation with others) [54]. Co-creation, or working as mediators for crowd acquisition, task specification, and evaluating outcomes, advocating, and co-development [70] are also motives for the crowd to participate in activities. Expectations [78] of earning profit such as “Cambros,” credit point, selling digital pictures online, voting things, scoring, commenting, decision-making, sharing reviews about a product [75] are also the reasons for crowd participation. Virtual teams complete tasks for bringing social capital [16, 17] (relationship) to their teams. The crowd also participates to brainstorm ideas and to agree about certain activities for alternative [3, 17] solutions [17, 39, 53, 60]; crowd workers are encouraged to participate as they are considered meaningful or relatedness as compared with others in the community [12]. The crowd contributes for detecting and correcting errors, and they desire to satisfy standards [21]. Membership, efficacy, reciprocity, etc., may be the reason for participation [20]. The reasons for crowd participation are represented in Table 8.Table 8
Presentation of various reasons for crowd participation.
S. no.Reasons of participationCitation1Honoraria[71]2Future employments/work[8, 35, 37, 43, 60, 71, 72, 77]3Enhancing performance[12, 71]4Money[8, 16, 23, 27, 29, 41, 43, 48, 52, 55, 56, 60, 61, 68, 77]5Payment[3, 4, 6, 7, 11, 13, 18, 19, 35, 37–39, 45, 64–66, 72, 73, 80]6Cash[50, 75]7Entertainment/fun/enjoyment[5, 6, 11, 12, 15, 16, 37, 41, 54, 60–62, 68, 70, 72, 78]8Skills development[6, 8, 15, 17, 22, 27, 41, 45, 47, 49, 61, 68, 77, 78]9Reputation[16, 17, 22, 35, 54, 65, 68, 77]10Gaining/sharing knowledge[6, 11, 15–17, 21, 49, 53, 60, 61, 70, 75, 78]11Prize/reward[36, 37, 49, 71, 78]12Funds collection[12, 49, 53, 75]13Finding answers to questions[3, 35, 49]14Inspiration[49]15Education/learning[6, 12, 15–17, 19, 20, 35, 37, 45, 47, 49, 60, 70, 72, 77]16Appreciation[49]17Incentives[11, 35, 37, 52]18Recognition[6, 11, 14, 37, 41, 52, 60–62, 75, 77]19Fame[37]20Gaining and sharing experiences[12, 17, 27, 39, 41, 53, 60, 62]21Altruism[12, 16, 38, 62, 77]22Rewards (monetary/nonmonetary, economical)[1, 3, 13, 14, 16, 17, 20, 25, 30, 41, 42, 45, 47, 54, 55, 59, 60, 62–64, 69, 70, 75, 77, 78, 83]23Friendship[16]24Socialization[5, 8, 11, 12, 16, 20, 54, 70]25Gaining and sharing idea/ideology[12, 17, 20, 38, 53, 60, 75, 80]26Self-need[38]27Self-development[38]28Authorship[38]29Increasing utility[18]30Extra bonus/income/points[12, 18, 23, 35, 71]31Compensation[16, 31, 61, 66]32Extrinsic and intrinsic rewards/needs[5, 6, 12, 14, 20, 21, 60, 67, 77]33Self esteem[6, 16, 78]34Voluntary participation[6, 10, 74]35Well-being/love of community, intellectual stimulation[2, 5, 6, 8, 10, 11, 16, 41, 77]36Communication[39, 53]37Cooperation[15, 20, 72]38Collaboration[12, 15, 49, 53]39Competition[15, 16, 61]40Remuneration[2, 19, 41, 56]41Career perspective[14, 19, 20, 41]42Provision of services[2, 40, 62, 80]43Sharing and acquiring information[40, 62]44Accomplishing task[12, 40, 41]45Self-marketability[41, 60, 77]46Pleasure[41]47Self-efficacy[20, 41, 54]48Contribution/sharing work[41, 60]49Earning extra income[43]50Providing or gathering feedback[47, 55]51Fee[57]52Enhancement of expertise level[59]53Integrative motivation[60]54Self-advertisement/self-expression[60, 61]55Individuals are free to performing tasks, i.e., task autonomy[61]56Social comparison[54]57Self-evaluation[54]58Sponsorship[61]59Achievements of goals[61]60For recommendation[62, 75]61Suggestion[62]62Financial benefits/rewards[6, 11, 22, 25, 41, 62, 63, 73, 78]63Ranking[62]64Solution[17, 39, 53, 60]65Tangible and intangible rewards[54]66Satisfaction[12, 21, 60, 61, 70]67Co-creation[70]68Co-development[70]69Crowd acquisition[70]70Advocating[70]71Specifying tasks[70]72Evaluating outcomes[70]73Earning profit[75]74Cambros[75]75Credit points[75]76Selling[75]77Voting[75]78Scoring[75]79Commenting[75]80Sharing reviews[75]81Decision-making[17, 75]82Gaining social capital[16, 17]83Brainstorming[17]84Finding alternative solutions[3, 17]85Altruistic contribution[9]86Financial and nonfinancial motives[7]87Immediate/delayed payoffs[5]88Importance[41]89Competence[8, 12, 41, 60, 61]90Best job performer[41]91Trusted[41]92Interest[16, 60, 61, 65]93Curiosity[12, 60, 77]94Self-affirmation[60]95Professional and personal relation[17, 60]96Self-belongingness[60]97Meaningfulness[12]98Relatedness[12]99Self-development[12]100Membership[20]101Reciprocity[20]102Visibility[7]103Task variety[61]104Task complexity[61]105Expectation[78]106Detect and correct errors[21]107Desire for satisfying standards[21]


### 3.4. Existing Techniques Used for Crowd Selection (RQ3)

#### 3.4.1. Profile-Based Searching

Profile [40] is an identity of crowd workers. In the registration process, crowd workers have to express themselves by posting personal profiles. Workers are selected for different projects [7] using profile data. Information about age, sex, and ability of workers to perform tasks is present in the profile [40]. Profile captures properties, sets, and describes participant name, location, etc., using ontology-defined vocabulary [82]. Workers’ attributes [68] and trust factor are present in the profile [34]. Personality-trait-based tool on various profiles is used to select workers [71]. Profiling will help in personalization of the intelligent decision and enhances the quality and selection of appropriate workers (crowd) for tasks. Personalization of crowd members would improve the decision quality [57].


#### 3.4.2. Selection on the Basis of Skill Assessment

Skills assessment is used for evaluation and certification of worker reliability to assist in the job matching process. Organization offers certification, which certifies that a worker has relevant skills [3, 7, 25, 31, 47, 49, 51, 52, 60, 76, 83]; these certifications are then used for recruiting purpose [51]. Workers who possess required skills are assigned tasks [3]. Skill evolution mechanism may be adopted for giving a chance to prove skills [52] Varieties of skills are needed from general, to specific, to situational [60]. Initial screening of the crowd is carried out based on skills [49] that are required for completing a task [25].


#### 3.4.3. Expertise Filtering

Expertise [3, 9, 13, 31, 46, 49, 58, 68, 71, 79] filtering is carried out to check the expertise level of workers. Expertise-estimation approaches estimate the expertise of workers [58]. The right crowd is selected based on their expertise level [68, 79]. Workers are assigned to a task if they have expertise in the required field [3]. As an individual has expertise in the task, he or she will try to complete it with full attention [31]. It depends on task relevance and diverse expertise, and the level of participant selected [46].


#### 3.4.4. Selection on the Basis of Crowd Attributes, Behaviors, and Attitudes

There are various attributes of crowd workers [68] which are significant for value match [31]. Attributes may be social [36], which shows crowd online or offline status, and social activities of workers assist in understanding routine behaviors [53]. Workers’ behavior is noticed when they complete tasks and bear a definite behavioral pattern [26]. Agent-based (AB) technique is applied to identify behaviors of crowd workers individually and to analyze the diverse attribute of participants. Workers are represented as agents and they have various characteristics. At platform, these agents arrive in nonhomogeneous passion distribution which are assigned with unique IDs based on their characteristics specified with utility factor which show their behaviors. An agent has a decision-making state that consists of two components, i.e., registration and submission of tasks. Agent decision is related to information that is received from agent’s community and social environment, and from other competing agents. The agent is responsible for submitting the task, and upon submission his or her attributes for reliability factor are updated [34]. The positivity of the crowd worker attitude increases their interest in solving a task [57].


#### 3.4.5. Selection on the Basis of Trustworthiness

Trust is the main factor for selecting workers for a task [3, 9, 29, 48]. A requestor may recruit trustworthy crowd workers [9]. For identifying trustworthy workers, SSC (Strong Social Component) and C-AWSA (Context-Aware Worker Selection Approach) is used, which is an efficient and valuable algorithm for selecting trustworthy workers. For optimization purposes, quality of trust as well as path utility is used. Forward searching Algorithm is used to calculate the trustworthiness of a worker [29]. Trust-Based Access Control (TBAC) strategy is utilized for computing trust value. Fuzzy Inference System evaluates the trust values and based on these values, access for a task is granted to crowd workers. The discrete model is used to decide whether an entity is trustworthy or not [48]. Trust factor of agent is updated when he or she submits a task [34]. “Career ladders” may be generated for trusted workers by organization for participating in high-level assignments [31]. Trustworthy workers are placed at a higher level [47].


#### 3.4.6. Selection on the Basis of Performance

Crowd workers are selected according to past performance [42] in various tasks [6, 68]. Worker previous task registration and winner records [49] are necessary for assigning tasks. The workers’ performances may increase with training [31]. For dealing with unknown worker performance in multiarmed bandit (MAB), initial exploration phase consistently samples the performance of workers using budget, and in the exploitation phase best workers are selected [56]. Affinity values can be calculated by participants’ social attributes and their previous task execution record [36]. Crowdsourcing platform is concerned with monitoring workers for evaluating and updating their skills depending on the quality of the tasks completed in the past [47]. December uses previous records of members of a crowd [81] from previous activities or social networks [82] for selection.


#### 3.4.7. Selection on the Basis of Feedback

The crowdsourcing platform gives feedback (temporary ranking) to crowd workers and based on these feedback, workers decides to quit or to compete for performing a task [3]. Feedback is increased or decreased while working [68].


#### 3.4.8. Role-Wise Selection

Roles assigned to crowd workers change with the change in the crowdsourcing environment [48]. Flash organization selects a crowd automatically, as they are structured in hierarchical form according to their roles and responsibilities [3].


#### 3.4.9. Selection on the Basis of Bid

Crowd workers are selected on the basis of winning bids. These workers make a bid based on their calculated effort and cost for getting a task. Only workers with winning bids are selected to submit a solution of the task [3]. Workers are selected statically, and after bidding winners are selected dynamically [50]. Workers with lower bids are hired first. Each new bid must satisfy the affinity and ability constraints. In the process, the winning bid gets the payment from the platform. For carrying out decision by platform, the users wait and stay online, and they may report dishonest time of entrance or exit in order to exploit their service. They may postpone their bid time and confer a prior exit time. During the time new challenges arise bid-independence, a user's payment cannot be affected by its bid, and the real appearance/exit time of the participant should not be known and is submitted truthfully. For the subsequent challenge, every active bid will contribute in the auction and their payment will be updated [36]. Classical bid auction was extended for ensuring quality [52].


#### 3.4.10. Selection Based on Job/Qualification Tests

Requesters specify requirements for workers to participate in activity. These requirements may consist of various tests that a participant has to complete in order to qualify for a job [68]. Workers are judged through qualification [3]or through job test [31]. Workers must pass qualification test before participating in tasks/projects [6]. Qualification test will ensure that worker has the knowledge for designing the user interface. Workers were selected based on the score achieved in the qualification test [4]. Requestor selects workers based on fulfilling some qualification requirement [66].


#### 3.4.11. Time- and Constraints-Wise Crowd Selection

Workers are encouraged to appear on time [50]. Crowds are registered, requirement of requestors are analyzed, time and duration is decided, and an appropriate crowd is selected [3]. Task agent announces an open call with instruction to available workers who will fulfill precise conditions. The task agent receives a reply from participants that are willing. A favorable set of workers is selected based on time constraints. Subsequently, it transfers a confirmation to the selected worker [40]. Requestors divide a complex task into subtasks, which increases the task probability of workers to accept the task in a specified time [1]. Participants are selected on the basis of timeliness of workers in prior accomplished tasks [9].


#### 3.4.12. Selection on the Basis of Belt Level

Workers are grouped in the registration order into five different belts, i.e., red, green, yellow, blue, and gray, which represent the skill level of workers. Reliability of workers is measured by registration and completion of the submitted task [44].


#### 3.4.13. Crowd Selection on the Basis of Experience Level

Crowd from a large pool of people are selected on the basis of experience [7, 45] of at least two years [24]. Experience strategy is developed for selecting experienced workers (testers) who have reported many bugs in the past [46]. Leader of a team is selected according to his or her previous experience [22].


#### 3.4.14. Ranking System Used for Crowd Selection

A ranking system was proposed, which dynamically modifies the worker skills. In ranking systems, trustworthy workers are placed at a higher level and others at a lower level. Workers are ranked in three categories, i.e., new comers, associates, and seniors. Ranking allows platforms to automatically find an appropriate worker for a specific task [47].


#### 3.4.15. Task Assignment Technique for Crowd Selection

For selection of qualified workers for sensing task, various task assignment techniques are applied. In MCS, worker selection is a challenging issue and it affects the sensing efficiency and quality. Various criteria are adopted for filtering the unsuitable worker. The task assignments framework consists of participants who use sensors for obtaining or measuring required data about his or her interested subject. Application/end users request with data through task and utilization of information required by participant, tasking entities distribute task to workers [62].


#### 3.4.16. Crowd Selection on the Basis of Capability/Ability

Organizations select employees who have the capability to produce great and diverse ideas for solving technical problems [71]. The organization makes a decision to pre-asses the ability of a crowd to perform complex tasks [66]. A worker is selected on the basis of ability [1]. The Borda ranking algorithm is utilized to check capability for selecting participants for the accomplishment of a dynamic task [23].


#### 3.4.17. Demographic Filtering for Crowd Selection

Crowd workers have different demographics [42, 46]. Demographic filtering selects specific country/location people [6, 68]; workers are selected according to their demography [22]. Workers provide information about demography if they are willing to participate in tasks [4].


#### 3.4.18. Crowd Selection on the Basis of Availability of Workers

Workers transfer messages to ensure their availability [40] or online status [36] on platform. Tasks are successively posted to a pool of workers [80]. Workers’ characteristics are identified for analyzing the availability of workers to perform and complete tasks [44]. Workers are assigned with tasks based on their availability [13].


#### 3.4.19. Recommendation-Based Crowd Selection

For a project, appropriate workers are recommended [10, 71]. The recommendation system is used to assess the crowd [28]. Workers are ranked according to recommendation levels [48]. For guiding workers to perform suitable tasks, recommendation techniques are used [49]. The recommendation system helps crowd workers in finding related tasks [39].


#### 3.4.20. Screening-Based Crowd Selection

Initial screening of the crowd is carried out based on skills that are required for completing a task [25]. Workers must pass an initial prescreening exam that consists of gold tasks. Prescreening assists requesters in inferring worker’s reliability [80].


#### 3.4.21. Relevant Background Filtering for Crowd Selection

A crowd is selected from diverse sources and with various backgrounds [60]. In some crowdsourcing activities, crowd workers’ recruitment is restricted to only workers who have declared that they have a relevant background [24].


#### 3.4.22. Crowd Selection on the Basis of Reputation

For checking the suitability of workers, their reputations are checked [9]. The reputation system calculates the reputation score of online workers based on collective ratings of employers that had hired them for tasks in the past [51]. A truthful online reputation updating algorithm was proposed for updating workers’ reputation [50]. Reputations of registered workers are checked before their participation in tasks. After the submission of tasks, quality is determined as it is used to control the reputation [65].


#### 3.4.23. Selection of Crowd on the Basis of Rating

Rating was used as important measure for generation of participant profiles. Workers are assigned only those tasks which were suitable for them [52]. Ratings assist in selecting an appropriate candidate for tasks [25].


#### 3.4.24. Crowd Selection on the Basis of Incentives/Reward Mechanisms

Incentive mechanism was proposed for worker selection. Workers are selected statically, and after bidding, winners are selected dynamically [50]. Participant incentives’ requirements are considered for selection [23]. Tasks are assigned to crowd based on associated reward [13].


#### 3.4.25. Crowd Selection on the Basis of Voting Scores

Task requester selects workers based on some criteria such as majority voting [26, 71].


#### 3.4.26. Crowd Selection According to Social Relation of Workers

A worker having high social context values is selected [29]. For distributing tasks in friends without getting their information, social relationship is utilized in the process. “A Task-Distributing System of Crowdsourcing Based on Social Relation Cognition, TDSRC”only accommodates interaction information of requestor and friends. Using social relationships, the model builds a trust chain among requestors and solvers, and thus increases the reliability of task distribution [83].


#### 3.4.27. Crowd Selection on the Basis of Auction

Organizations utilize auction-based model for the selection of crowd workers based on their skills, qualification, and trustworthiness [3]. Classical bid auction was extended for ensuring quality. Appropriate workers are invited and the bids are ranked for specific rewards [52].


#### 3.4.28. Crowd Selection on the Basis of Matching Mechanism

Matching of task to appropriate workers should help the crowd to spend their strength in tasks to benefit the crowd as well as organization [31]. Task and tester matching technique is used to select a crowd (testers) from a large pool of people [45]. Project managers use a matching method to select the leader of a team [22]. The existing techniques for selecting an appropriate crowd for tasks are represented in Table 9.Table 9
Existing technique used for crowd selection.
S. no.DescriptionExisting techniquesCitation1Workers are selected according to personal profiles that contain various attributes such as sex, age, workers’ ability, demography, expertise, experience, past performance, etc.Profiling-based selection[7, 39, 40, 42, 47, 49, 52, 57, 59, 68, 69, 71, 76, 81, 82]2For accomplishment of tasks, crowds are selected according to their skills related to various tasks. Skills test assessments are carried out to certify that workers have the necessary skills for completing tasks.Skill assessments[3, 7, 25, 31, 47, 49, 51, 52, 60, 76, 83]3Expertise-based selection is carried out to capture experts to perform specific tasks.Expertise filtering[3, 9, 13, 31, 46, 49, 58, 68, 71, 79]4Behavior and attributes are important characteristics of workers participating in various tasks. Workers are selected on the basis of their attitude and behaviors observed in performing various tasks.Selection on the basis of crowd attributes, behaviors, and attitudes[26, 31, 34, 36, 53, 57, 68]5Trust is an important factor for the selection of the appropriate worker for tasks. Workers are assigned with different trust values and they are selected based on these trust values.Selection on the basis of trustworthiness[3, 9, 29, 31, 34, 47, 48]6Individuals are selected based on their previous or current performance in different types of tasks.Performance[6, 36, 42, 47, 49, 56, 68, 81, 82]7Feedback is provided to rank workers for competing or quitting various tasks; crowds are selected on the basis of various feedback provided by third parties, i.e., organizations or other participating bodies.Feedback mechanism[3, 68]8Workers are selected on the basis of their roles.Role-wise selection[3, 48]9Workers are allowed for making their bid, and appropriate workers are selected according to their bid level to perform a task. The bids are associated with specific rewards.Selection on the basis of bid[3, 36, 50, 52]10Organization conducts qualification- or job-specific tests to assess and select workers for various types of tasks. Qualification test ensures that workers have relevant knowledge for the task.Selection based on job/qualification tests[3, 4, 6, 31, 66, 68]11Optimal set of workers are selected based on time and cost constraints. Workers are selected on the basis of timeline followed in previous completed tasks.Time and constraints[1, 3, 9, 40, 50]12Workers are grouped in registration order into five different belts, i.e., red, green, yellow, blue, and gray, that represent the skill level of workers.Selection on the basis of belt level[44]13Crowd participants are selected on the basis of their experience level.Crowd selection on the basis of experience level[7, 22, 24, 45, 46]14For worker selection, ranking is applied. Workers are classified into three categories, i.e., new comers, associates, and seniors. Ranking allows platforms to automatically find out an appropriate worker for a specific task.Ranking[47]15In mobile crowd sensing, task assignment techniques are used to select crowd workers. The task assignments’ framework is used to capture participants who use sensors for obtaining or measuring specific data.Task assignment technique[62]16A worker is selected on the basis of his or ability and capability toward a task.Crowd selection on the basis of capability/Ability[1, 23, 66, 71]17Candidates are selected according to their country of origin.Demographic filtering[4, 6, 22, 42, 68]18Before allocation of task to workers, their availability on platforms is observed.Crowd selection on the basis of availability of workers[13, 36, 40, 44, 80]19For assessing the crowd capabilities, a recommendation system was utilized. Crowd workers must have access to the recommendation system to help them in finding related tasks according to their profiles.Recommendation-based crowd selection[10, 28, 39, 48, 49, 71]20Screening of the crowd is carried out based on skills that are required for completing a task. Workers are screened out from a diverse group by conducting tests, etc.Screening-based crowd selection[25, 80]21Crowd workers’ recruitment is restricted to only workers who have declared that they have a background to a specific task and if they possess a working history of at least two years.Relevant background filtering for crowd selection[24, 60]22The reputation system is used to select participants on the basis of their level of reputation.Crowd selection on the basis of reputation[9, 50, 51, 65]23The crowd is rated based on the quality of responses. Rating assists in selecting the appropriate crowd for tasks. Workers are selected by job seekers according to their rating.Selection of crowd on the basis of rating[25, 52]24An incentive mechanism was proposed for worker selection. Workers are selected statically; after bidding, winners are selected dynamicallyCrowd selection on the basis of incentives/reward mechanisms[13, 23, 50]25The task requester selects workers based on some criteria, such as majority voting, for estimating correct answers of a task.Crowd selection on the basis of voting scores[26, 71]26Social relationship among requestors and solvers is established and the task is given to friends of a friend; due to this, the reliability of task distribution is increased.Crowd selection according to social relation of workers[29, 83]27Matches complex tasks requiring multiple skills to appropriate workers. Workers are invited for making a bid for a task, and this bidding was based on price as well as worker profile.Crowd selection on the basis of auction[3, 52]28To select a crowd (testers) from a large pool of people, the technique was used. The tasks are matched with crowd and only the appropriate crowd is permitted.Crowd selection on the basis of matching mechanism[22, 31, 45]


## 3.1. Overview of Selected Papers

For the proposed study, 81 Research studies are selected on the basis of QoA criteria. Among these studies, 55 papers were published in journal articles, 1 paper in book section, 20 are extracted from conference proceedings, and 5 studies were retrieved from conference papers. The total number of selected papers and their percentages are presented in Figure6; the selected papers based on publication year are shown in Figure 7.Figure 6
Percentages of selected studies.Figure 7
Number of selected papers based on publication year.


## 3.2. Significant Features of Crowd in Crowdsourcing (RQ1)

Organizations are interested in leveraging and gaining the knowledge of people. Superior techniques are applied for collecting this knowledge from external experts for enhancing the performance of products. Managers of an organization who are interested in solving challenges must select specialized professionals or subject experts from the external world who have the knowledge for solving a new organizational challenge. These specialized persons represent a solid approach to increase the absorptive capacity or the company’s capability to identify, assimilate, and exploit external knowledge. Various approaches are used by specialized professionals for increasing the storage capacity, or they may use the potential of the company for identifying, selecting, assimilating, and utilizing the knowledge of experts [71]. In crowdsourcing, the entire project or a part of it is outsourced to a large pool [39], of distributive and cost effective’s labors [72]. Crowd workers possess some characteristics, i.e., they can be identified with certain rules that direct their behaviors, they do not depend on others, they control their actions, and they may be flexible [34, 81] in changing their behavior with the changing environment [34]. Crowd workers may be newcomers (novices) [12, 20, 41, 47]; anonymous participants [3, 6, 7, 11, 19, 73, 83]; inexperienced [11], fraudulent [1], incompetent [21] workers; heterogeneous workers [38, 52, 56, 64] in working styles; unknown workers [73]; transient solvers (it greatly influences the quality of a task) [83]; undefined workers [3]; nonprofessionals [49, 56, 83]; nonexperts [49, 68]; untrained participants [3, 35]; untrustworthy workers only interested in gaining the rewards that are associated with tasks and thus do not work sincerely, and this is a negative factor of the crowd [29]. They act selfishly (negative feature) for utility utilization [18]. Workers may act maliciously [35, 80, 83],which limits the quality of tasks, or may be “eager beavers” [35] who outperform appropriate workers [5]. Trainees may accomplish tasks [3, 41] in the crowdsourcing activity. The crowd may be autonomous [34], fast [2] unique [69], appropriate [3], right [28, 79], reliable/efficient [2, 25, 72, 73, 80], loyal [70], truthful [36], trustworthy [29, 42, 44, 50, 63] who complete assigned tasks sincerely. Workers are coordinative, adaptable [16] (as they can change themselves with the change in the work environment), “Energetic” having energy [10, 17], capable [1, 42, 60] of performing tasks, and are “creative” as they creatively [67, 68, 78] perform different types of work. They are competent workers [16, 39, 63] who utilize common sense [68] in various tasks. Crowds are smart [25, 27], educated/qualified [1, 13, 42], professionals [71, 75], having skills (skillful workers) [2, 8, 10, 12, 14, 17, 26, 28, 37, 48, 51, 54, 56, 57, 59–61, 67, 76–78], expertise [58, 59, 81, 82], experience [14, 17, 19, 24, 43, 46, 47, 53, 76] who possess knowledge [2, 8, 11, 13–15, 17, 19, 22, 30, 35, 38, 49, 55, 60, 61, 68, 70, 71, 75, 76] for getting a task done [35]. Workers may be cooperative [12], collaborative [21, 39, 76]. Crowds may work as volunteers [10, 15, 17, 74–76] for solving large problems [5, 7, 21, 54, 60, 64, 68, 75, 78], i.e., they are solution providers [5]. Workers can make decisions [15].The various features of the crowd are represented in Table 6 and Table 7.Table 6
Negative features/characteristics of crowd workers.
S. no.Negative featuresCitations1Nonprofessionals[49, 56, 83]2Nonexperts[49, 68]3Untrustworthy[29]4Unskilled workers[48]5Anonymous[3, 6, 7, 11, 19, 73, 83]6Untrained participants[3, 35]7Malicious workers[35, 80, 83]8Selfish workers[18]9Novice[12, 20, 41, 47]10Transient workers[83]11Inexperience[11]12Fraudulent[1]13Incompetence[21]Table 7
Positive features/characteristics of crowd workers.
S. no.Positive featuresCitation1Professionals[71, 75]2Expertise[58, 59, 81, 82]3Trustworthy[29, 42, 44, 50, 63]4Skill[2, 8, 10, 12, 14, 17, 26, 28, 37, 48, 51, 54, 56, 57, 59–61, 67, 76–78]5Identifiable[34]6Autonomous[34]7Flexible[34, 81]8Creative[67, 68, 78]9Common sense[68]10Knowledge[2, 8, 11, 13–15, 17, 19, 22, 30, 35, 38, 49, 55, 60, 61, 68, 70, 71, 75, 76]11Appropriate[3]12Eager beavers[35]13Trainee[3, 41]14Competent[16, 39, 63]15Adaptable[16]16Coordinative[16]17Heterogeneous[38, 52, 56, 64]18Collaborative[21, 39, 76]19Decision maker[15]20Qualified/educated[1, 13, 42]21Ability[1, 42, 60]22Problem-solving[5, 7, 21, 54, 60, 64, 68, 75, 78]23Unique (workers)[69]24Volunteers[10, 15, 17, 74–76]25Energetic[10, 17]26Experienced[14, 17, 19, 24, 43, 46, 47, 53, 76]27Reliable/efficient (workers)[2, 25, 72, 73, 80]28Accurate[74]29Smart[25, 27]30Cost-effective workers[72]31Solution providers[5]32Right[28, 79]33Fast[2]34Cooperative[12]35Truthful[36]36Inventive[64]37Loyal[70]


## 3.3. Reasons behind Crowd Participation (RQ2)

Various incentive measures are adopted to motivate participant for contribution like for teams in organizations, pointing system is adopted and participants are provided with different points depending on their performance level. These points are then converted into prizes and gifts. External participants are provided with honoraria and likelihood of career prospects/advancement [14, 19, 20, 41] and being employed in future work [71]. Various workers complete different types of tasks such as sentiment analysis and tagging of image, solving microprogramming tasks in order to receive benefits of remuneration [2, 19, 41, 56] financial [6], monetary [41, 47, 83], reward [37, 78], prize [36, 37, 49, 71, 78], payment [45, 65, 66] like immediate payoffs (making money) [5], cash [50, 75], fee [57], direct compensation [16, 31, 61, 66] and money [8, 55], extra bonuses [12, 35], extra income [43], non-monetary awards including delayed payoffs (the potential to take freelance work) [5], status [3], pleasure personal enjoyment/fun [5, 62], and for learning purpose [6, 45] to increase their knowledge [15–17], increase reputation [65, 77] in public display by increase in point and badges. The motivation may be an introjected one in which crowd participates for recognition [6, 62] among peers, fame [37]. People also compete with each other to develop their skills [6, 45]. Crowds are rewarded based on the quality of the completed task. A worker who produces quality results is awarded with incentives [35, 37] and access for future work [8, 43]. Worker observes developers [15] work, raise, and collecting funds [12, 49, 53, 75] for projects. Crowd also participates for inspiration [49] from other solutions and revision of solutions for generating better alternative [3, 17] for existing solutions. Communication [39, 53] with each other to manage their time [39] is also a reason for participation. Crowd works in cooperation [15, 20] and in collaboration [12, 53] for improving products. Crowd participants may also participate to provide ideas [12, 53] for various developmental projects. A participant participates as he/she perceives curiosity [12, 60, 77], self-affirmation [60], self-estimation [6, 78], and appreciation [49] of requester organizations. Workers are motivated for self-marketability [41, 77], visibility [7]. Self-efficacy [20, 41, 54] which constitutes four types of incentives, namely, accomplishment [12, 40, 41] (workers feel that they are accomplishing something while participating in tasks), importance (workers observe that their doing is important to others), competence [41] (working on crowdsourcing tasks provide a wisdom of competency), and doing best job (workers desire to know that they are doing best), and to contribute. There are four incentives regarding contribution, namely, effectiveness (how effective the tasks are to others), helping others (participant completes the task to help other people), trust (workers are trusted by organizations), and supporting others (participants support each other in the accomplishment of tasks) [41]. The crowd also participates to find an appropriate answer for a question [3, 35, 49]. An individual can be motivated to gain or share experience [12, 62] and increase his or her expertise level [59]. Other reasons for motivating the crowd to participate in tasks are volunteering (free contribution) [6, 10, 74], reciprocity and expectancy in which users participate for intellectual stimulation [2]. To help a community as they expect that others will also contribute in society [2, 41, 77], contribution of each every one will help in solving a community problem. Crowd contributes altruistically [9] without greediness/expectations for something; they do it out of sympathy, which is known as altruism [62]. There are also some factors that motivate the crowd for participation like making new friends [16] and for socialization [54, 70] purposes; people also contribute in crowdsourcing as they have interest in [16, 60, 65] a field. Developers for task are motivated by ideology, self-need, self-development, and for preserving authorship of their own work to get paid to provide best solutions to problems [39]. Crowds may provide a service [2, 40], acquire information [40, 62], or may accomplish a task like booking a table in a restaurant [40]. Contestants also participate so that organizations provide them feedback [47, 55] on their completed work. The crowds are motivated by means of intrinsic and extrinsic motivation [77] Intrinsic motivation is concerned with participation that increases interest and joy due to which participant feels competency, fulfillment, and satisfaction [12, 61]. Extrinsic motivation is working on something that provides some distinguishable results. It involves engagement because of external incentives such as compensation [66]. The participation may be due to identifiable motivation, which is a reaction of freedom and violation since the deeds are concerned with identity and personal goal achievements [61]. Integrative motives [60] may be a cause of participation in which participant activities are considered significant and meaningful. The crowd is also motivated by networking motives for building professional or personal relation [17, 60]. This will increase self-belonging [60] and increase engagement. Self-marketing [41, 60, 77] or self-advertisement [60, 61] is a major motivation for participation in projects for individuals looking for a job. Workers perceive that competition provides workers with a number of benefits, and they have to show eagerness to engage in competitions [15, 16, 61]. Workers are willing to participate as they are provided with task autonomy [61]; participants are offered a high level of control on their actions in competition. Task variety is another motive for participation where a participant is allowed to provide solutions from different viewpoints. Players are allowed to use their abilities and skills for developing solutions. Task complexity is another motive for participation as task is complex; its completion requires high competency, skills. Workers are interested in performing complex tasks to gain self-expression and a sense of competency. The crowd also participates for sponsoring an organization [61]. The crowd also shares and acquires information [40, 62], recommends [62, 75] some things, shares life experiences, provides suggestion, and increases the level of ranking [62]. Crowd teams participate in crowd source task for receiving tangible (monetary) and intangible (reputation) rewards, for social comparison (self-evaluation with others) [54]. Co-creation, or working as mediators for crowd acquisition, task specification, and evaluating outcomes, advocating, and co-development [70] are also motives for the crowd to participate in activities. Expectations [78] of earning profit such as “Cambros,” credit point, selling digital pictures online, voting things, scoring, commenting, decision-making, sharing reviews about a product [75] are also the reasons for crowd participation. Virtual teams complete tasks for bringing social capital [16, 17] (relationship) to their teams. The crowd also participates to brainstorm ideas and to agree about certain activities for alternative [3, 17] solutions [17, 39, 53, 60]; crowd workers are encouraged to participate as they are considered meaningful or relatedness as compared with others in the community [12]. The crowd contributes for detecting and correcting errors, and they desire to satisfy standards [21]. Membership, efficacy, reciprocity, etc., may be the reason for participation [20]. The reasons for crowd participation are represented in Table 8.Table 8
Presentation of various reasons for crowd participation.
S. no.Reasons of participationCitation1Honoraria[71]2Future employments/work[8, 35, 37, 43, 60, 71, 72, 77]3Enhancing performance[12, 71]4Money[8, 16, 23, 27, 29, 41, 43, 48, 52, 55, 56, 60, 61, 68, 77]5Payment[3, 4, 6, 7, 11, 13, 18, 19, 35, 37–39, 45, 64–66, 72, 73, 80]6Cash[50, 75]7Entertainment/fun/enjoyment[5, 6, 11, 12, 15, 16, 37, 41, 54, 60–62, 68, 70, 72, 78]8Skills development[6, 8, 15, 17, 22, 27, 41, 45, 47, 49, 61, 68, 77, 78]9Reputation[16, 17, 22, 35, 54, 65, 68, 77]10Gaining/sharing knowledge[6, 11, 15–17, 21, 49, 53, 60, 61, 70, 75, 78]11Prize/reward[36, 37, 49, 71, 78]12Funds collection[12, 49, 53, 75]13Finding answers to questions[3, 35, 49]14Inspiration[49]15Education/learning[6, 12, 15–17, 19, 20, 35, 37, 45, 47, 49, 60, 70, 72, 77]16Appreciation[49]17Incentives[11, 35, 37, 52]18Recognition[6, 11, 14, 37, 41, 52, 60–62, 75, 77]19Fame[37]20Gaining and sharing experiences[12, 17, 27, 39, 41, 53, 60, 62]21Altruism[12, 16, 38, 62, 77]22Rewards (monetary/nonmonetary, economical)[1, 3, 13, 14, 16, 17, 20, 25, 30, 41, 42, 45, 47, 54, 55, 59, 60, 62–64, 69, 70, 75, 77, 78, 83]23Friendship[16]24Socialization[5, 8, 11, 12, 16, 20, 54, 70]25Gaining and sharing idea/ideology[12, 17, 20, 38, 53, 60, 75, 80]26Self-need[38]27Self-development[38]28Authorship[38]29Increasing utility[18]30Extra bonus/income/points[12, 18, 23, 35, 71]31Compensation[16, 31, 61, 66]32Extrinsic and intrinsic rewards/needs[5, 6, 12, 14, 20, 21, 60, 67, 77]33Self esteem[6, 16, 78]34Voluntary participation[6, 10, 74]35Well-being/love of community, intellectual stimulation[2, 5, 6, 8, 10, 11, 16, 41, 77]36Communication[39, 53]37Cooperation[15, 20, 72]38Collaboration[12, 15, 49, 53]39Competition[15, 16, 61]40Remuneration[2, 19, 41, 56]41Career perspective[14, 19, 20, 41]42Provision of services[2, 40, 62, 80]43Sharing and acquiring information[40, 62]44Accomplishing task[12, 40, 41]45Self-marketability[41, 60, 77]46Pleasure[41]47Self-efficacy[20, 41, 54]48Contribution/sharing work[41, 60]49Earning extra income[43]50Providing or gathering feedback[47, 55]51Fee[57]52Enhancement of expertise level[59]53Integrative motivation[60]54Self-advertisement/self-expression[60, 61]55Individuals are free to performing tasks, i.e., task autonomy[61]56Social comparison[54]57Self-evaluation[54]58Sponsorship[61]59Achievements of goals[61]60For recommendation[62, 75]61Suggestion[62]62Financial benefits/rewards[6, 11, 22, 25, 41, 62, 63, 73, 78]63Ranking[62]64Solution[17, 39, 53, 60]65Tangible and intangible rewards[54]66Satisfaction[12, 21, 60, 61, 70]67Co-creation[70]68Co-development[70]69Crowd acquisition[70]70Advocating[70]71Specifying tasks[70]72Evaluating outcomes[70]73Earning profit[75]74Cambros[75]75Credit points[75]76Selling[75]77Voting[75]78Scoring[75]79Commenting[75]80Sharing reviews[75]81Decision-making[17, 75]82Gaining social capital[16, 17]83Brainstorming[17]84Finding alternative solutions[3, 17]85Altruistic contribution[9]86Financial and nonfinancial motives[7]87Immediate/delayed payoffs[5]88Importance[41]89Competence[8, 12, 41, 60, 61]90Best job performer[41]91Trusted[41]92Interest[16, 60, 61, 65]93Curiosity[12, 60, 77]94Self-affirmation[60]95Professional and personal relation[17, 60]96Self-belongingness[60]97Meaningfulness[12]98Relatedness[12]99Self-development[12]100Membership[20]101Reciprocity[20]102Visibility[7]103Task variety[61]104Task complexity[61]105Expectation[78]106Detect and correct errors[21]107Desire for satisfying standards[21]


## 3.4. Existing Techniques Used for Crowd Selection (RQ3)

### 3.4.1. Profile-Based Searching

Profile [40] is an identity of crowd workers. In the registration process, crowd workers have to express themselves by posting personal profiles. Workers are selected for different projects [7] using profile data. Information about age, sex, and ability of workers to perform tasks is present in the profile [40]. Profile captures properties, sets, and describes participant name, location, etc., using ontology-defined vocabulary [82]. Workers’ attributes [68] and trust factor are present in the profile [34]. Personality-trait-based tool on various profiles is used to select workers [71]. Profiling will help in personalization of the intelligent decision and enhances the quality and selection of appropriate workers (crowd) for tasks. Personalization of crowd members would improve the decision quality [57].


### 3.4.2. Selection on the Basis of Skill Assessment

Skills assessment is used for evaluation and certification of worker reliability to assist in the job matching process. Organization offers certification, which certifies that a worker has relevant skills [3, 7, 25, 31, 47, 49, 51, 52, 60, 76, 83]; these certifications are then used for recruiting purpose [51]. Workers who possess required skills are assigned tasks [3]. Skill evolution mechanism may be adopted for giving a chance to prove skills [52] Varieties of skills are needed from general, to specific, to situational [60]. Initial screening of the crowd is carried out based on skills [49] that are required for completing a task [25].


### 3.4.3. Expertise Filtering

Expertise [3, 9, 13, 31, 46, 49, 58, 68, 71, 79] filtering is carried out to check the expertise level of workers. Expertise-estimation approaches estimate the expertise of workers [58]. The right crowd is selected based on their expertise level [68, 79]. Workers are assigned to a task if they have expertise in the required field [3]. As an individual has expertise in the task, he or she will try to complete it with full attention [31]. It depends on task relevance and diverse expertise, and the level of participant selected [46].


### 3.4.4. Selection on the Basis of Crowd Attributes, Behaviors, and Attitudes

There are various attributes of crowd workers [68] which are significant for value match [31]. Attributes may be social [36], which shows crowd online or offline status, and social activities of workers assist in understanding routine behaviors [53]. Workers’ behavior is noticed when they complete tasks and bear a definite behavioral pattern [26]. Agent-based (AB) technique is applied to identify behaviors of crowd workers individually and to analyze the diverse attribute of participants. Workers are represented as agents and they have various characteristics. At platform, these agents arrive in nonhomogeneous passion distribution which are assigned with unique IDs based on their characteristics specified with utility factor which show their behaviors. An agent has a decision-making state that consists of two components, i.e., registration and submission of tasks. Agent decision is related to information that is received from agent’s community and social environment, and from other competing agents. The agent is responsible for submitting the task, and upon submission his or her attributes for reliability factor are updated [34]. The positivity of the crowd worker attitude increases their interest in solving a task [57].


### 3.4.5. Selection on the Basis of Trustworthiness

Trust is the main factor for selecting workers for a task [3, 9, 29, 48]. A requestor may recruit trustworthy crowd workers [9]. For identifying trustworthy workers, SSC (Strong Social Component) and C-AWSA (Context-Aware Worker Selection Approach) is used, which is an efficient and valuable algorithm for selecting trustworthy workers. For optimization purposes, quality of trust as well as path utility is used. Forward searching Algorithm is used to calculate the trustworthiness of a worker [29]. Trust-Based Access Control (TBAC) strategy is utilized for computing trust value. Fuzzy Inference System evaluates the trust values and based on these values, access for a task is granted to crowd workers. The discrete model is used to decide whether an entity is trustworthy or not [48]. Trust factor of agent is updated when he or she submits a task [34]. “Career ladders” may be generated for trusted workers by organization for participating in high-level assignments [31]. Trustworthy workers are placed at a higher level [47].


### 3.4.6. Selection on the Basis of Performance

Crowd workers are selected according to past performance [42] in various tasks [6, 68]. Worker previous task registration and winner records [49] are necessary for assigning tasks. The workers’ performances may increase with training [31]. For dealing with unknown worker performance in multiarmed bandit (MAB), initial exploration phase consistently samples the performance of workers using budget, and in the exploitation phase best workers are selected [56]. Affinity values can be calculated by participants’ social attributes and their previous task execution record [36]. Crowdsourcing platform is concerned with monitoring workers for evaluating and updating their skills depending on the quality of the tasks completed in the past [47]. December uses previous records of members of a crowd [81] from previous activities or social networks [82] for selection.


### 3.4.7. Selection on the Basis of Feedback

The crowdsourcing platform gives feedback (temporary ranking) to crowd workers and based on these feedback, workers decides to quit or to compete for performing a task [3]. Feedback is increased or decreased while working [68].


### 3.4.8. Role-Wise Selection

Roles assigned to crowd workers change with the change in the crowdsourcing environment [48]. Flash organization selects a crowd automatically, as they are structured in hierarchical form according to their roles and responsibilities [3].


### 3.4.9. Selection on the Basis of Bid

Crowd workers are selected on the basis of winning bids. These workers make a bid based on their calculated effort and cost for getting a task. Only workers with winning bids are selected to submit a solution of the task [3]. Workers are selected statically, and after bidding winners are selected dynamically [50]. Workers with lower bids are hired first. Each new bid must satisfy the affinity and ability constraints. In the process, the winning bid gets the payment from the platform. For carrying out decision by platform, the users wait and stay online, and they may report dishonest time of entrance or exit in order to exploit their service. They may postpone their bid time and confer a prior exit time. During the time new challenges arise bid-independence, a user's payment cannot be affected by its bid, and the real appearance/exit time of the participant should not be known and is submitted truthfully. For the subsequent challenge, every active bid will contribute in the auction and their payment will be updated [36]. Classical bid auction was extended for ensuring quality [52].


### 3.4.10. Selection Based on Job/Qualification Tests

Requesters specify requirements for workers to participate in activity. These requirements may consist of various tests that a participant has to complete in order to qualify for a job [68]. Workers are judged through qualification [3]or through job test [31]. Workers must pass qualification test before participating in tasks/projects [6]. Qualification test will ensure that worker has the knowledge for designing the user interface. Workers were selected based on the score achieved in the qualification test [4]. Requestor selects workers based on fulfilling some qualification requirement [66].


### 3.4.11. Time- and Constraints-Wise Crowd Selection

Workers are encouraged to appear on time [50]. Crowds are registered, requirement of requestors are analyzed, time and duration is decided, and an appropriate crowd is selected [3]. Task agent announces an open call with instruction to available workers who will fulfill precise conditions. The task agent receives a reply from participants that are willing. A favorable set of workers is selected based on time constraints. Subsequently, it transfers a confirmation to the selected worker [40]. Requestors divide a complex task into subtasks, which increases the task probability of workers to accept the task in a specified time [1]. Participants are selected on the basis of timeliness of workers in prior accomplished tasks [9].


### 3.4.12. Selection on the Basis of Belt Level

Workers are grouped in the registration order into five different belts, i.e., red, green, yellow, blue, and gray, which represent the skill level of workers. Reliability of workers is measured by registration and completion of the submitted task [44].


### 3.4.13. Crowd Selection on the Basis of Experience Level

Crowd from a large pool of people are selected on the basis of experience [7, 45] of at least two years [24]. Experience strategy is developed for selecting experienced workers (testers) who have reported many bugs in the past [46]. Leader of a team is selected according to his or her previous experience [22].


### 3.4.14. Ranking System Used for Crowd Selection

A ranking system was proposed, which dynamically modifies the worker skills. In ranking systems, trustworthy workers are placed at a higher level and others at a lower level. Workers are ranked in three categories, i.e., new comers, associates, and seniors. Ranking allows platforms to automatically find an appropriate worker for a specific task [47].


### 3.4.15. Task Assignment Technique for Crowd Selection

For selection of qualified workers for sensing task, various task assignment techniques are applied. In MCS, worker selection is a challenging issue and it affects the sensing efficiency and quality. Various criteria are adopted for filtering the unsuitable worker. The task assignments framework consists of participants who use sensors for obtaining or measuring required data about his or her interested subject. Application/end users request with data through task and utilization of information required by participant, tasking entities distribute task to workers [62].


### 3.4.16. Crowd Selection on the Basis of Capability/Ability

Organizations select employees who have the capability to produce great and diverse ideas for solving technical problems [71]. The organization makes a decision to pre-asses the ability of a crowd to perform complex tasks [66]. A worker is selected on the basis of ability [1]. The Borda ranking algorithm is utilized to check capability for selecting participants for the accomplishment of a dynamic task [23].


### 3.4.17. Demographic Filtering for Crowd Selection

Crowd workers have different demographics [42, 46]. Demographic filtering selects specific country/location people [6, 68]; workers are selected according to their demography [22]. Workers provide information about demography if they are willing to participate in tasks [4].


### 3.4.18. Crowd Selection on the Basis of Availability of Workers

Workers transfer messages to ensure their availability [40] or online status [36] on platform. Tasks are successively posted to a pool of workers [80]. Workers’ characteristics are identified for analyzing the availability of workers to perform and complete tasks [44]. Workers are assigned with tasks based on their availability [13].


### 3.4.19. Recommendation-Based Crowd Selection

For a project, appropriate workers are recommended [10, 71]. The recommendation system is used to assess the crowd [28]. Workers are ranked according to recommendation levels [48]. For guiding workers to perform suitable tasks, recommendation techniques are used [49]. The recommendation system helps crowd workers in finding related tasks [39].


### 3.4.20. Screening-Based Crowd Selection

Initial screening of the crowd is carried out based on skills that are required for completing a task [25]. Workers must pass an initial prescreening exam that consists of gold tasks. Prescreening assists requesters in inferring worker’s reliability [80].


### 3.4.21. Relevant Background Filtering for Crowd Selection

A crowd is selected from diverse sources and with various backgrounds [60]. In some crowdsourcing activities, crowd workers’ recruitment is restricted to only workers who have declared that they have a relevant background [24].


### 3.4.22. Crowd Selection on the Basis of Reputation

For checking the suitability of workers, their reputations are checked [9]. The reputation system calculates the reputation score of online workers based on collective ratings of employers that had hired them for tasks in the past [51]. A truthful online reputation updating algorithm was proposed for updating workers’ reputation [50]. Reputations of registered workers are checked before their participation in tasks. After the submission of tasks, quality is determined as it is used to control the reputation [65].


### 3.4.23. Selection of Crowd on the Basis of Rating

Rating was used as important measure for generation of participant profiles. Workers are assigned only those tasks which were suitable for them [52]. Ratings assist in selecting an appropriate candidate for tasks [25].


### 3.4.24. Crowd Selection on the Basis of Incentives/Reward Mechanisms

Incentive mechanism was proposed for worker selection. Workers are selected statically, and after bidding, winners are selected dynamically [50]. Participant incentives’ requirements are considered for selection [23]. Tasks are assigned to crowd based on associated reward [13].


### 3.4.25. Crowd Selection on the Basis of Voting Scores

Task requester selects workers based on some criteria such as majority voting [26, 71].


### 3.4.26. Crowd Selection According to Social Relation of Workers

A worker having high social context values is selected [29]. For distributing tasks in friends without getting their information, social relationship is utilized in the process. “A Task-Distributing System of Crowdsourcing Based on Social Relation Cognition, TDSRC”only accommodates interaction information of requestor and friends. Using social relationships, the model builds a trust chain among requestors and solvers, and thus increases the reliability of task distribution [83].


### 3.4.27. Crowd Selection on the Basis of Auction

Organizations utilize auction-based model for the selection of crowd workers based on their skills, qualification, and trustworthiness [3]. Classical bid auction was extended for ensuring quality. Appropriate workers are invited and the bids are ranked for specific rewards [52].


### 3.4.28. Crowd Selection on the Basis of Matching Mechanism

Matching of task to appropriate workers should help the crowd to spend their strength in tasks to benefit the crowd as well as organization [31]. Task and tester matching technique is used to select a crowd (testers) from a large pool of people [45]. Project managers use a matching method to select the leader of a team [22]. The existing techniques for selecting an appropriate crowd for tasks are represented in Table 9.Table 9
Existing technique used for crowd selection.
S. no.DescriptionExisting techniquesCitation1Workers are selected according to personal profiles that contain various attributes such as sex, age, workers’ ability, demography, expertise, experience, past performance, etc.Profiling-based selection[7, 39, 40, 42, 47, 49, 52, 57, 59, 68, 69, 71, 76, 81, 82]2For accomplishment of tasks, crowds are selected according to their skills related to various tasks. Skills test assessments are carried out to certify that workers have the necessary skills for completing tasks.Skill assessments[3, 7, 25, 31, 47, 49, 51, 52, 60, 76, 83]3Expertise-based selection is carried out to capture experts to perform specific tasks.Expertise filtering[3, 9, 13, 31, 46, 49, 58, 68, 71, 79]4Behavior and attributes are important characteristics of workers participating in various tasks. Workers are selected on the basis of their attitude and behaviors observed in performing various tasks.Selection on the basis of crowd attributes, behaviors, and attitudes[26, 31, 34, 36, 53, 57, 68]5Trust is an important factor for the selection of the appropriate worker for tasks. Workers are assigned with different trust values and they are selected based on these trust values.Selection on the basis of trustworthiness[3, 9, 29, 31, 34, 47, 48]6Individuals are selected based on their previous or current performance in different types of tasks.Performance[6, 36, 42, 47, 49, 56, 68, 81, 82]7Feedback is provided to rank workers for competing or quitting various tasks; crowds are selected on the basis of various feedback provided by third parties, i.e., organizations or other participating bodies.Feedback mechanism[3, 68]8Workers are selected on the basis of their roles.Role-wise selection[3, 48]9Workers are allowed for making their bid, and appropriate workers are selected according to their bid level to perform a task. The bids are associated with specific rewards.Selection on the basis of bid[3, 36, 50, 52]10Organization conducts qualification- or job-specific tests to assess and select workers for various types of tasks. Qualification test ensures that workers have relevant knowledge for the task.Selection based on job/qualification tests[3, 4, 6, 31, 66, 68]11Optimal set of workers are selected based on time and cost constraints. Workers are selected on the basis of timeline followed in previous completed tasks.Time and constraints[1, 3, 9, 40, 50]12Workers are grouped in registration order into five different belts, i.e., red, green, yellow, blue, and gray, that represent the skill level of workers.Selection on the basis of belt level[44]13Crowd participants are selected on the basis of their experience level.Crowd selection on the basis of experience level[7, 22, 24, 45, 46]14For worker selection, ranking is applied. Workers are classified into three categories, i.e., new comers, associates, and seniors. Ranking allows platforms to automatically find out an appropriate worker for a specific task.Ranking[47]15In mobile crowd sensing, task assignment techniques are used to select crowd workers. The task assignments’ framework is used to capture participants who use sensors for obtaining or measuring specific data.Task assignment technique[62]16A worker is selected on the basis of his or ability and capability toward a task.Crowd selection on the basis of capability/Ability[1, 23, 66, 71]17Candidates are selected according to their country of origin.Demographic filtering[4, 6, 22, 42, 68]18Before allocation of task to workers, their availability on platforms is observed.Crowd selection on the basis of availability of workers[13, 36, 40, 44, 80]19For assessing the crowd capabilities, a recommendation system was utilized. Crowd workers must have access to the recommendation system to help them in finding related tasks according to their profiles.Recommendation-based crowd selection[10, 28, 39, 48, 49, 71]20Screening of the crowd is carried out based on skills that are required for completing a task. Workers are screened out from a diverse group by conducting tests, etc.Screening-based crowd selection[25, 80]21Crowd workers’ recruitment is restricted to only workers who have declared that they have a background to a specific task and if they possess a working history of at least two years.Relevant background filtering for crowd selection[24, 60]22The reputation system is used to select participants on the basis of their level of reputation.Crowd selection on the basis of reputation[9, 50, 51, 65]23The crowd is rated based on the quality of responses. Rating assists in selecting the appropriate crowd for tasks. Workers are selected by job seekers according to their rating.Selection of crowd on the basis of rating[25, 52]24An incentive mechanism was proposed for worker selection. Workers are selected statically; after bidding, winners are selected dynamicallyCrowd selection on the basis of incentives/reward mechanisms[13, 23, 50]25The task requester selects workers based on some criteria, such as majority voting, for estimating correct answers of a task.Crowd selection on the basis of voting scores[26, 71]26Social relationship among requestors and solvers is established and the task is given to friends of a friend; due to this, the reliability of task distribution is increased.Crowd selection according to social relation of workers[29, 83]27Matches complex tasks requiring multiple skills to appropriate workers. Workers are invited for making a bid for a task, and this bidding was based on price as well as worker profile.Crowd selection on the basis of auction[3, 52]28To select a crowd (testers) from a large pool of people, the technique was used. The tasks are matched with crowd and only the appropriate crowd is permitted.Crowd selection on the basis of matching mechanism[22, 31, 45]


## 3.4.1. Profile-Based Searching

Profile [40] is an identity of crowd workers. In the registration process, crowd workers have to express themselves by posting personal profiles. Workers are selected for different projects [7] using profile data. Information about age, sex, and ability of workers to perform tasks is present in the profile [40]. Profile captures properties, sets, and describes participant name, location, etc., using ontology-defined vocabulary [82]. Workers’ attributes [68] and trust factor are present in the profile [34]. Personality-trait-based tool on various profiles is used to select workers [71]. Profiling will help in personalization of the intelligent decision and enhances the quality and selection of appropriate workers (crowd) for tasks. Personalization of crowd members would improve the decision quality [57].


## 3.4.2. Selection on the Basis of Skill Assessment

Skills assessment is used for evaluation and certification of worker reliability to assist in the job matching process. Organization offers certification, which certifies that a worker has relevant skills [3, 7, 25, 31, 47, 49, 51, 52, 60, 76, 83]; these certifications are then used for recruiting purpose [51]. Workers who possess required skills are assigned tasks [3]. Skill evolution mechanism may be adopted for giving a chance to prove skills [52] Varieties of skills are needed from general, to specific, to situational [60]. Initial screening of the crowd is carried out based on skills [49] that are required for completing a task [25].


## 3.4.3. Expertise Filtering

Expertise [3, 9, 13, 31, 46, 49, 58, 68, 71, 79] filtering is carried out to check the expertise level of workers. Expertise-estimation approaches estimate the expertise of workers [58]. The right crowd is selected based on their expertise level [68, 79]. Workers are assigned to a task if they have expertise in the required field [3]. As an individual has expertise in the task, he or she will try to complete it with full attention [31]. It depends on task relevance and diverse expertise, and the level of participant selected [46].


## 3.4.4. Selection on the Basis of Crowd Attributes, Behaviors, and Attitudes

There are various attributes of crowd workers [68] which are significant for value match [31]. Attributes may be social [36], which shows crowd online or offline status, and social activities of workers assist in understanding routine behaviors [53]. Workers’ behavior is noticed when they complete tasks and bear a definite behavioral pattern [26]. Agent-based (AB) technique is applied to identify behaviors of crowd workers individually and to analyze the diverse attribute of participants. Workers are represented as agents and they have various characteristics. At platform, these agents arrive in nonhomogeneous passion distribution which are assigned with unique IDs based on their characteristics specified with utility factor which show their behaviors. An agent has a decision-making state that consists of two components, i.e., registration and submission of tasks. Agent decision is related to information that is received from agent’s community and social environment, and from other competing agents. The agent is responsible for submitting the task, and upon submission his or her attributes for reliability factor are updated [34]. The positivity of the crowd worker attitude increases their interest in solving a task [57].


## 3.4.5. Selection on the Basis of Trustworthiness

Trust is the main factor for selecting workers for a task [3, 9, 29, 48]. A requestor may recruit trustworthy crowd workers [9]. For identifying trustworthy workers, SSC (Strong Social Component) and C-AWSA (Context-Aware Worker Selection Approach) is used, which is an efficient and valuable algorithm for selecting trustworthy workers. For optimization purposes, quality of trust as well as path utility is used. Forward searching Algorithm is used to calculate the trustworthiness of a worker [29]. Trust-Based Access Control (TBAC) strategy is utilized for computing trust value. Fuzzy Inference System evaluates the trust values and based on these values, access for a task is granted to crowd workers. The discrete model is used to decide whether an entity is trustworthy or not [48]. Trust factor of agent is updated when he or she submits a task [34]. “Career ladders” may be generated for trusted workers by organization for participating in high-level assignments [31]. Trustworthy workers are placed at a higher level [47].


## 3.4.6. Selection on the Basis of Performance

Crowd workers are selected according to past performance [42] in various tasks [6, 68]. Worker previous task registration and winner records [49] are necessary for assigning tasks. The workers’ performances may increase with training [31]. For dealing with unknown worker performance in multiarmed bandit (MAB), initial exploration phase consistently samples the performance of workers using budget, and in the exploitation phase best workers are selected [56]. Affinity values can be calculated by participants’ social attributes and their previous task execution record [36]. Crowdsourcing platform is concerned with monitoring workers for evaluating and updating their skills depending on the quality of the tasks completed in the past [47]. December uses previous records of members of a crowd [81] from previous activities or social networks [82] for selection.


## 3.4.7. Selection on the Basis of Feedback

The crowdsourcing platform gives feedback (temporary ranking) to crowd workers and based on these feedback, workers decides to quit or to compete for performing a task [3]. Feedback is increased or decreased while working [68].


## 3.4.8. Role-Wise Selection

Roles assigned to crowd workers change with the change in the crowdsourcing environment [48]. Flash organization selects a crowd automatically, as they are structured in hierarchical form according to their roles and responsibilities [3].


## 3.4.9. Selection on the Basis of Bid

Crowd workers are selected on the basis of winning bids. These workers make a bid based on their calculated effort and cost for getting a task. Only workers with winning bids are selected to submit a solution of the task [3]. Workers are selected statically, and after bidding winners are selected dynamically [50]. Workers with lower bids are hired first. Each new bid must satisfy the affinity and ability constraints. In the process, the winning bid gets the payment from the platform. For carrying out decision by platform, the users wait and stay online, and they may report dishonest time of entrance or exit in order to exploit their service. They may postpone their bid time and confer a prior exit time. During the time new challenges arise bid-independence, a user's payment cannot be affected by its bid, and the real appearance/exit time of the participant should not be known and is submitted truthfully. For the subsequent challenge, every active bid will contribute in the auction and their payment will be updated [36]. Classical bid auction was extended for ensuring quality [52].


## 3.4.10. Selection Based on Job/Qualification Tests

Requesters specify requirements for workers to participate in activity. These requirements may consist of various tests that a participant has to complete in order to qualify for a job [68]. Workers are judged through qualification [3]or through job test [31]. Workers must pass qualification test before participating in tasks/projects [6]. Qualification test will ensure that worker has the knowledge for designing the user interface. Workers were selected based on the score achieved in the qualification test [4]. Requestor selects workers based on fulfilling some qualification requirement [66].


## 3.4.11. Time- and Constraints-Wise Crowd Selection

Workers are encouraged to appear on time [50]. Crowds are registered, requirement of requestors are analyzed, time and duration is decided, and an appropriate crowd is selected [3]. Task agent announces an open call with instruction to available workers who will fulfill precise conditions. The task agent receives a reply from participants that are willing. A favorable set of workers is selected based on time constraints. Subsequently, it transfers a confirmation to the selected worker [40]. Requestors divide a complex task into subtasks, which increases the task probability of workers to accept the task in a specified time [1]. Participants are selected on the basis of timeliness of workers in prior accomplished tasks [9].


## 3.4.12. Selection on the Basis of Belt Level

Workers are grouped in the registration order into five different belts, i.e., red, green, yellow, blue, and gray, which represent the skill level of workers. Reliability of workers is measured by registration and completion of the submitted task [44].


## 3.4.13. Crowd Selection on the Basis of Experience Level

Crowd from a large pool of people are selected on the basis of experience [7, 45] of at least two years [24]. Experience strategy is developed for selecting experienced workers (testers) who have reported many bugs in the past [46]. Leader of a team is selected according to his or her previous experience [22].


## 3.4.14. Ranking System Used for Crowd Selection

A ranking system was proposed, which dynamically modifies the worker skills. In ranking systems, trustworthy workers are placed at a higher level and others at a lower level. Workers are ranked in three categories, i.e., new comers, associates, and seniors. Ranking allows platforms to automatically find an appropriate worker for a specific task [47].


## 3.4.15. Task Assignment Technique for Crowd Selection

For selection of qualified workers for sensing task, various task assignment techniques are applied. In MCS, worker selection is a challenging issue and it affects the sensing efficiency and quality. Various criteria are adopted for filtering the unsuitable worker. The task assignments framework consists of participants who use sensors for obtaining or measuring required data about his or her interested subject. Application/end users request with data through task and utilization of information required by participant, tasking entities distribute task to workers [62].


## 3.4.16. Crowd Selection on the Basis of Capability/Ability

Organizations select employees who have the capability to produce great and diverse ideas for solving technical problems [71]. The organization makes a decision to pre-asses the ability of a crowd to perform complex tasks [66]. A worker is selected on the basis of ability [1]. The Borda ranking algorithm is utilized to check capability for selecting participants for the accomplishment of a dynamic task [23].


## 3.4.17. Demographic Filtering for Crowd Selection

Crowd workers have different demographics [42, 46]. Demographic filtering selects specific country/location people [6, 68]; workers are selected according to their demography [22]. Workers provide information about demography if they are willing to participate in tasks [4].


## 3.4.18. Crowd Selection on the Basis of Availability of Workers

Workers transfer messages to ensure their availability [40] or online status [36] on platform. Tasks are successively posted to a pool of workers [80]. Workers’ characteristics are identified for analyzing the availability of workers to perform and complete tasks [44]. Workers are assigned with tasks based on their availability [13].


## 3.4.19. Recommendation-Based Crowd Selection

For a project, appropriate workers are recommended [10, 71]. The recommendation system is used to assess the crowd [28]. Workers are ranked according to recommendation levels [48]. For guiding workers to perform suitable tasks, recommendation techniques are used [49]. The recommendation system helps crowd workers in finding related tasks [39].


## 3.4.20. Screening-Based Crowd Selection

Initial screening of the crowd is carried out based on skills that are required for completing a task [25]. Workers must pass an initial prescreening exam that consists of gold tasks. Prescreening assists requesters in inferring worker’s reliability [80].


## 3.4.21. Relevant Background Filtering for Crowd Selection

A crowd is selected from diverse sources and with various backgrounds [60]. In some crowdsourcing activities, crowd workers’ recruitment is restricted to only workers who have declared that they have a relevant background [24].


## 3.4.22. Crowd Selection on the Basis of Reputation

For checking the suitability of workers, their reputations are checked [9]. The reputation system calculates the reputation score of online workers based on collective ratings of employers that had hired them for tasks in the past [51]. A truthful online reputation updating algorithm was proposed for updating workers’ reputation [50]. Reputations of registered workers are checked before their participation in tasks. After the submission of tasks, quality is determined as it is used to control the reputation [65].


## 3.4.23. Selection of Crowd on the Basis of Rating

Rating was used as important measure for generation of participant profiles. Workers are assigned only those tasks which were suitable for them [52]. Ratings assist in selecting an appropriate candidate for tasks [25].


## 3.4.24. Crowd Selection on the Basis of Incentives/Reward Mechanisms

Incentive mechanism was proposed for worker selection. Workers are selected statically, and after bidding, winners are selected dynamically [50]. Participant incentives’ requirements are considered for selection [23]. Tasks are assigned to crowd based on associated reward [13].


## 3.4.25. Crowd Selection on the Basis of Voting Scores

Task requester selects workers based on some criteria such as majority voting [26, 71].


## 3.4.26. Crowd Selection According to Social Relation of Workers

A worker having high social context values is selected [29]. For distributing tasks in friends without getting their information, social relationship is utilized in the process. “A Task-Distributing System of Crowdsourcing Based on Social Relation Cognition, TDSRC”only accommodates interaction information of requestor and friends. Using social relationships, the model builds a trust chain among requestors and solvers, and thus increases the reliability of task distribution [83].


## 3.4.27. Crowd Selection on the Basis of Auction

Organizations utilize auction-based model for the selection of crowd workers based on their skills, qualification, and trustworthiness [3]. Classical bid auction was extended for ensuring quality. Appropriate workers are invited and the bids are ranked for specific rewards [52].


## 3.4.28. Crowd Selection on the Basis of Matching Mechanism

Matching of task to appropriate workers should help the crowd to spend their strength in tasks to benefit the crowd as well as organization [31]. Task and tester matching technique is used to select a crowd (testers) from a large pool of people [45]. Project managers use a matching method to select the leader of a team [22]. The existing techniques for selecting an appropriate crowd for tasks are represented in Table 9.Table 9
Existing technique used for crowd selection.
S. no.DescriptionExisting techniquesCitation1Workers are selected according to personal profiles that contain various attributes such as sex, age, workers’ ability, demography, expertise, experience, past performance, etc.Profiling-based selection[7, 39, 40, 42, 47, 49, 52, 57, 59, 68, 69, 71, 76, 81, 82]2For accomplishment of tasks, crowds are selected according to their skills related to various tasks. Skills test assessments are carried out to certify that workers have the necessary skills for completing tasks.Skill assessments[3, 7, 25, 31, 47, 49, 51, 52, 60, 76, 83]3Expertise-based selection is carried out to capture experts to perform specific tasks.Expertise filtering[3, 9, 13, 31, 46, 49, 58, 68, 71, 79]4Behavior and attributes are important characteristics of workers participating in various tasks. Workers are selected on the basis of their attitude and behaviors observed in performing various tasks.Selection on the basis of crowd attributes, behaviors, and attitudes[26, 31, 34, 36, 53, 57, 68]5Trust is an important factor for the selection of the appropriate worker for tasks. Workers are assigned with different trust values and they are selected based on these trust values.Selection on the basis of trustworthiness[3, 9, 29, 31, 34, 47, 48]6Individuals are selected based on their previous or current performance in different types of tasks.Performance[6, 36, 42, 47, 49, 56, 68, 81, 82]7Feedback is provided to rank workers for competing or quitting various tasks; crowds are selected on the basis of various feedback provided by third parties, i.e., organizations or other participating bodies.Feedback mechanism[3, 68]8Workers are selected on the basis of their roles.Role-wise selection[3, 48]9Workers are allowed for making their bid, and appropriate workers are selected according to their bid level to perform a task. The bids are associated with specific rewards.Selection on the basis of bid[3, 36, 50, 52]10Organization conducts qualification- or job-specific tests to assess and select workers for various types of tasks. Qualification test ensures that workers have relevant knowledge for the task.Selection based on job/qualification tests[3, 4, 6, 31, 66, 68]11Optimal set of workers are selected based on time and cost constraints. Workers are selected on the basis of timeline followed in previous completed tasks.Time and constraints[1, 3, 9, 40, 50]12Workers are grouped in registration order into five different belts, i.e., red, green, yellow, blue, and gray, that represent the skill level of workers.Selection on the basis of belt level[44]13Crowd participants are selected on the basis of their experience level.Crowd selection on the basis of experience level[7, 22, 24, 45, 46]14For worker selection, ranking is applied. Workers are classified into three categories, i.e., new comers, associates, and seniors. Ranking allows platforms to automatically find out an appropriate worker for a specific task.Ranking[47]15In mobile crowd sensing, task assignment techniques are used to select crowd workers. The task assignments’ framework is used to capture participants who use sensors for obtaining or measuring specific data.Task assignment technique[62]16A worker is selected on the basis of his or ability and capability toward a task.Crowd selection on the basis of capability/Ability[1, 23, 66, 71]17Candidates are selected according to their country of origin.Demographic filtering[4, 6, 22, 42, 68]18Before allocation of task to workers, their availability on platforms is observed.Crowd selection on the basis of availability of workers[13, 36, 40, 44, 80]19For assessing the crowd capabilities, a recommendation system was utilized. Crowd workers must have access to the recommendation system to help them in finding related tasks according to their profiles.Recommendation-based crowd selection[10, 28, 39, 48, 49, 71]20Screening of the crowd is carried out based on skills that are required for completing a task. Workers are screened out from a diverse group by conducting tests, etc.Screening-based crowd selection[25, 80]21Crowd workers’ recruitment is restricted to only workers who have declared that they have a background to a specific task and if they possess a working history of at least two years.Relevant background filtering for crowd selection[24, 60]22The reputation system is used to select participants on the basis of their level of reputation.Crowd selection on the basis of reputation[9, 50, 51, 65]23The crowd is rated based on the quality of responses. Rating assists in selecting the appropriate crowd for tasks. Workers are selected by job seekers according to their rating.Selection of crowd on the basis of rating[25, 52]24An incentive mechanism was proposed for worker selection. Workers are selected statically; after bidding, winners are selected dynamicallyCrowd selection on the basis of incentives/reward mechanisms[13, 23, 50]25The task requester selects workers based on some criteria, such as majority voting, for estimating correct answers of a task.Crowd selection on the basis of voting scores[26, 71]26Social relationship among requestors and solvers is established and the task is given to friends of a friend; due to this, the reliability of task distribution is increased.Crowd selection according to social relation of workers[29, 83]27Matches complex tasks requiring multiple skills to appropriate workers. Workers are invited for making a bid for a task, and this bidding was based on price as well as worker profile.Crowd selection on the basis of auction[3, 52]28To select a crowd (testers) from a large pool of people, the technique was used. The tasks are matched with crowd and only the appropriate crowd is permitted.Crowd selection on the basis of matching mechanism[22, 31, 45]


## 4. Research Findings

The crowdsourcing activity involves allocation of tasks to the Internet crowd [2, 39]; with an open call, the crowd is involved in solving tasks that are complex. These crowds are employed from web-enabled communities. Crowd workers represent some attributes, such as qualification, age, gender, language, worker location, skills, past service, and experience. Workers are selected on the basis of these attributes [13, 17, 40, 45]. Crowdsourcing assists workers or an organization to seek services, contents, ideas, or services from a huge group of people online (crowd) [80]. The crowd workers are dynamic and creative, with a variety of motivations and experiences. Unique crowd workers’ participation is required in crowdsourcing. Global workers have to represent themselves as experts by posting personalized profiles created during the process of registration [7, 69]. Information of workers, such as sex, age, and ability, is present in the worker profile [40]. The crowd is evaluated on the basis of the work performed [22]. An appropriate set of workers is selected for performing tasks [46]. For achieving quality outcomes, an employer should carefully select workers [56]. The quality provided by workers depends on the distribution of tasks to the members of the crowd, which requires appropriate mechanisms to be controlled such as by screening workers [13, 42]. Initial screening of the crowd is carried out on the basis of skills required to complete a task. Care must be taken while selecting the crowd, as it is noticed that the quality improves with a diverse crowd selection [25]. Quality outcomes produced by crowd workers depends on certain aspects such as worker skills, their experience regarding tasks, and their commitment for performing tasks [47].


## 5. Discussion

The analysis highlights the point of view behind research in the area of crowdsourcing and provides a synopsis of current crowd selection techniques based on certain aspects. Crowd selection is an important phase in crowdsourcing. With the help of selection methodologies, the appropriate participant for a task is selected. The review has a number of implications in the crowdsourcing activity. In the crowdsourcing activity, the task is performed by crowd workers. The study highlights the multi features of an online crowd in order to choose the right candidate for a task from Internet communities. The review also underscores as to why crowds participate in crowdsourcing tasks? Various techniques were suggested in literature for crowd selection. They are represented in this review for the purpose of understanding how the crowd was selected previously and to identify the measures for this selection process.


## 6. Validation Threats

The biased nature of publications and extraction of inaccurate data are considered as major threats regarding review protocol. The studies are selected on the basis of search string, described in the search strategy, which includes literature resources, selection on the basis of criteria, and quality assessment criteria. Search string relevant to specific formulated questions was used for the extraction of relevant studies. However, there might also be some irrelevant studies that are extracted using search strings. A selection criterion was adopted in compliance with formulated questions to extract relevant studies. Finally, the studies are evaluated on the basis of the quality of assessment, and these papers are analyzed thoroughly to minimize the threats linked with the extraction of inaccurate data.


## 7. Conclusion

In the proposed study, we addressed the problem of crowd selection in crowdsourcing. One of the key contributions of our research is to illustrate the various multiple features of the crowd. Another contribution of the review is to identify the various reasons of crowd participation, and the third contribution of this study is to capture various selection methods used for crowd selection. Research aims are extracted from various studies in the literature. The aims are extracted from online data repositories such as IEEE, ACM, ScienceDirect, Taylor & Francis, Wiley, Hindawi, and Springer. They were searched on the basis of search strings to extract relevant studies.Our preliminary objective was to capture the current state of crowd selection in crowdsourcing research and to assist developers and researchers to highlight what has been done in the literature. The review points out the multi features of the crowd, reasons for participation of the crowd for engagement in various tasks, and the different techniques used for the selection of an appropriate candidate for tasks. A systematic review was conducted, and a total of 81 relevant studies are selected based on QoA. Subsequent to study selection, on the basis of title or abstract, the studies were properly reviewed for the purpose of acknowledging that the study answers minimally two research questions or not.Our finding suggests that crowd selection is an important phase in the crowdsourcing activity; without proper selection of the crowd, the crowdsourcing activity will not be productive. This review will be a base study for future crowdsourcing researchers, as it highlights and explains the entity “crowd,” their feature, and their effectiveness in crowdsourcing activities. The study will also be useful to the organization if they want to hire/select appropriate workers for a task. The survey can also benefit platforms, as it provides them with insight features and the motivation of crowd workers.


## 8. Research Limitations

The review was conducted rigorously within the limits of specified questions. As a result, 81 related studies have been listed and chosen to be able to answer at least two formulated questions effectively. The related studies are chosen from 2010 to 2020. With the paradigm shift and the complex nature of the crowd and crowd selection, however, we cannot fully guarantee that all possible studies in the proposed research have been captured. Another point of concern is that, because only papers published in English were included in this study, significant or appropriate research published in non-English journals was overlooked. Assessment was carried out for research biasness due to the strong tendency of reporting biases of the precise strength and weakness of current techniques. However, in this review, the selected studies did not include gray literature, technical reports, work in progress, and unpublished work, which may be capable of answering any research question. Title and content of various studies were analyzed in order to ensure that these studies are capable of answering the formulated research questions. The assessment criteria used in this analysis cannot guarantee that those criteria are sufficient to identify the appropriate studies.


## 9. Future Research Suggestions

Crowd selection is an important phase in the crowdsourcing activity; without appropriate selection of crowd workers, the crowdsourcing activity is aimless. In the literature, the crowd workers are selected on the basis of only a single feature, such as trust, performance, reliability, accurateness, experience, knowledge, skills, expertise, etc. As no multi-criteria features have been used in the past for crowd selection, multi-criteria-based crowd selection is mandatory for the success of the crowdsourcing activity.



---

*Source: 9368128-2021-07-08.xml*