# An Efficient Decision Support System for the Selection of Appropriate Crowd in Crowdsourcing

Authors: Yongjun Huang; Shah Nazir; Jiyu Wu; Fida Hussain Khoso; Farhad Ali; Habib Ullah Khan
Journal: Complexity
 (2021)

---

## Abstract

Crowdsourcing is a complex task-solving model that utilizes humans for solving organizational specific problems. For assigning a crowdsourced task to an online crowd, crowd selection is carried out to select appropriate crowd for achieving the task. The efficiency and effectiveness of crowdsourcing may fail if irrelevant crowd is selected for performing a task. Early decisions regarding selection of a crowd can ultimately lead to successful completion of tasks. To select most appropriate crowd from crowdsourcing, this paper presents a decision support system (DSS) for appropriate selection of crowd. The system has been implemented in the Superdecision tool by plotting hierarchy of goals, criteria, and alternatives. Various calculations have been done for performing the proposed research. Results of the study reveal that the proposed system is effective and efficient for selection of crowd in crowdsourcing by performing various pairwise computation of the study.


---

## Body

## 1. Introduction

Crowds are online people who have abilities to accomplish different types of tasks. These crowds may be newcomers who are accomplishing tasks for the first time or they may be experienced members who have completed various tasks previously. Crowdsourcing is a practice that acquires the services of huge group of people for obtaining information or completing a project [1]. It is internet-enabled collaborative activity that solves organizational problems by collecting the knowledge of online communities. The contributing editor Jeff Howe in June 2006 first used the word “crowdsourcing” in article “The Rise of Crowdsourcing” that was published in Wired magazine [2]. “Crowdsourcing is a type of participative online activity in which individual, institution, non-profit organization, or company proposes to a group of individuals of varying knowledge, heterogeneity, and number, via a flexible open call, voluntary undertaking of a task. The undertaking of task, of variable complexity and modularity, and in which crowd should participate bringing their work, money, knowledge and/or experience, always entails mutual benefit [3]. The user will receive satisfaction of a given type of need, be it economic, social recognition, self-esteem, or the development of individual skills, while crowdsourcer will obtain and utilize to their advantage that what the user has brought to the venture, whose form will depend on type of activity undertaken” [4].The applications of crowdsourcing are used widely for software testing [5], usability testing [6], machine learning processes [7], and decision making [8]. The productivity of large organizations has been enhanced by crowdsourcing [9]. The crowds comprise diverse-background participants who possess skills relevant to tasks and experience in the field and have expertise in carrying out crowdsourced task or tackling complex problems [10]. Organizations are commonly using crowdsourcing to address challenges simultaneously with the large involvement of crowds. Crowdsourcing is an effective way to mitigate organizational dilemmas [11].Crowdsourcing helps business organizations to recruit global, cheap, and skilled workers from different platforms [12, 13]. The new era of Web 3.0 is driven by innovations in ICT and social networking, and as a result organizational decision-making process has also been changed [14]. Modern corporations use the Internet to recruit a massive crowd. The Internet is a media for contact between crowds and businesses, and they work together using gadgets like iPads, mobile phones, laptops, wearable watches, etc. [15–17]. The crowds are recruited for completing different tasks from social or global societies [18]. By consuming small amount of management cost, time organization can achieve appropriate solutions with multiple crowd worker participation [19, 20]. As crowdsourcing is an online activity, it may entail certain risks, such as the announcement of tasks on websites and the selection of a suitable and qualified team [21]. The increased interest of crowdsourcing makes the selection of crowd workers a challenge. The crowded workers may be untrustworthy whose work can be followed by different errors. The choice of right and proper workers would boost the efficacy of crowdsourcing [22, 23]. Different business organization employs a suitable worker to complete task [24]. The following are the contributions of the proposed study:(i)
A DSS is presented for the appropriate selection of crowd(ii)
The proposed system is implemented in the Superdecision tool(iii)
The hierarchy of goals, criteria, and alternatives is plotted with various pairwise comparisons to perform the proposed research(iv)
Results of the study reveal that the proposed approach is effective and efficientThe organization of the paper is as follows. Section2 presents the related work on the various aspects of the crowd and crowdsource concepts. Section 3 shows the details of the methodology with a description of the decision support system and the selection of features from literature. Results and discussion are given in Section 4. The paper is concluded in Section 5.


## 2. Literature Review

Crowdsourcing is an online process that could be linked to various challenges, such as crowd selection problem [21]. Various strategies, approaches, and models were presented in the past to address the crowd selection problem. Selection of crowd was based on the characteristics they possess which includes personal characteristics such as gender, age, qualification, education, language, and worker nationality; behavioral characteristics such as sociolects, left/right handed, and personality traits; cognitive or perceptual characteristics that include a person’s memory capacity, vision, hearing, or these may include skills, capabilities, past service, expertise, experience, and majors. Based on these characteristics, crowd are selected [25, 26]. A crowd targeting framework was implemented that automatically discovers and targets a specific crowd to improve data quality. The targeted crowd is selected by the worker characteristics such as nationality, education level, gender, and major or even personality test score and any other screening measures. Information gain that is a new characteristic measure for worker selection is also introduced. The framework selects workers using 3 main stages. The first is the probing stage in which the tasks are distributed to the whole population of crowd. Crowds are allowed to complete these tasks. The workers characteristics such as gender and location are also gathered from their profile for future use in this stage. The second is a discovery stage that is related to the discovery of the best workers, where unbiased worker samples of the entire crowd population are identified. The workers are evaluated using criteria such as good, bad, available, etc. The third is targeting stage in which the remaining and upcoming tasks are assigned to the discovered groups. The targeting stage improves the quality of data and increases budget performance [27].Workers are selected by various organizations based on their capabilities for generating ideas or solving problems related to technology [28]. To allow a worker to participate in difficult tasks, an organization assesses the worker’s ability [29]. Based on ability, a worker is selected [17]. For verifying workers ability, Borda ranking algorithm can be utilized [30]. Worker skills may also be an indicator for its selection as the skills reflect its ability to perform a task. The crowds are judged on the basis of their skills. Skills are, therefore, one of the main considerations for selection of right participants [31]. Workers possess various skills such as writing, IT, problem solving, process management, time management, communication, creative, e-skills, business thinking, and enterprise [32]. Skills’ assessment or testing is used to assess various worker skills and these are helpful in the task matching processes. Organization offers certification that does certify workers posse’s sufficient skills [33–35]. The certification is used for selection of workers [36]. The crowd who possess essential skills complete the task [33]. Trust is a major factor for consideration of workers for a task [23, 33].Organization selects crowd workers for accomplishment of various task based on its trustworthiness [37]. For evaluating trust value “Trust-Based Access Control (TBAC)” model is utilized, and for decision concerning whether a worker is to be trusted or not, a discrete model was implemented [38]. Crowd trust was proposed that is a context-aware model for the evaluation of trust related to the type of task “TaTrust” and for the calculation of trust associated with task reward amount “RaTrust.” For the selection of trustworthy workers with 2 context-aware trusts, “MOWS GA” that is an evolutionary algorithm and depends on NSGA-II was introduced. The dishonest workers can be identified using the crowd trust model [39]. A recruitment process was introduced in spatial mobile crowdsourcing that automatically selects trustworthy workers by utilizing the services of IoT. A huge group of workers is reduced to potential trustworthy workers using Lovain Community detection algorithm, and the optimal set of crowd is selected by utilizing integer Linear program [40]. By utilizing approaches of machine learning, the prediction of trustworthiness was improved with the exploration of endorsement (interwork relationship) [41].Workers may also be selected on the basis of their experience with tasks. Experience is considered as a crucial factor [31] for crowd selection. The crowd consists of huge masses of people and according to the level of experience best workers are selected [25, 42]. For selection of experienced participant for task, experience strategy is utilized [24]. Selection of crowd greatly relies on its expertise [24]. A crowd is selected based on its expertise level [43]. Only workers having requisite expertise are allowed to carry out the task [33]. For ensuring the workers expertise level filtering [28, 33] is performed. Workers are judged according to varied expertise using expertise-estimation approaches [44]. The task can considerately be performed by worker having expertise [45]. Qualified workers are judged by means of qualification tests and these assessments are superior filters for quality enhancement. The work quality can be controlled by conducting qualification tests. In these tests, a worker has to answer various questions provided by organizations. Workers must pass the qualification test before engaging with projects or tasks [46]. Workers are assessed based on their qualification level [33].Profile based selection is also carried out for worker selection as the profile represents the personal features of the worker that can be directly observed. The profile contains worker details such as sex, age, education, and history of accomplished tasks [31, 47]. Exploiting workers’ profiles would improve the assessment, assignments, and the quality of task [48]. Workers are responsible for maintaining and modifying their profiles for getting work from organizations [32]. For selection of workers based on workers profile personality based tool may be utilized [28]. Profile based approach was implemented for an effective selection of worker in crowdsourcing to reduce overhead time and budget by replacing an offline learning process with the online probing stage. This was done for the purpose of learning profile features and these features will be used by the online targeting algorithm for the selection of effective workers for different tasks [49]. The profile based selection of crowd can enhance the decision-making process of crowd selection [50].


## 3. Methodology

DSS is related to the discipline of information system area that supports and enhances the decision-making process of an organization [51]. It is difficult for decision makers to give preferences as high volumes of data regarding crowds are available. DSS is implemented for broadening the capabilities of human information and for enhancing the process of decision making when dealing with large amount of data [52]. Crowdsourcing can play a role in the organizational decision-making process. A complex problem can easily be solved by crowd as they provide ideas, solicit opinions, give prediction, accumulate knowledge, etc. [53]. There is a lack of research which suggests a DSS for the selection of suitable crowds. Existing research studies were analyzed for the purpose of identifying the multifeatures of crowd. Table 1 represents these features. The multifeatures will be used by our DSS for the appropriate selection of crowd. Crowdsourcing activity entails three entities that are crowdsourcer/requestor which are organizations, individuals, or institutions who initiate the crowdsourcing process and seek out the ability of people for completing tasks which are shown in Figure 1 [68]; the crowd that consists of large group of people having enactive, cognitive, and perceptual abilities for solving tasks [69]; and the platform or market which is an online website or place where workers acquire and accomplish tasks [70].Table 1
Features of crowd.
S. no.FeaturesCitation1Professionals[28]2Trustworthy[23, 49, 54]3Skill[10, 34, 55, 56]4Competent[54, 57, 58]5Collaborative[59, 60]6Decision maker[61]7Qualified/educated[17, 49, 62]8Problem solving[34, 42, 43, 56, 63–67]9Experienced[13, 18]Figure 1
Entities of crowdsourcing.The reason behind choosing the DSS for the proposed study was to consider the early decision of the crowd from the crowdsource. Various features of crowd were identified in the literature. Keeping in view the suitability of the crowd, the following key features were identified as the most suitable features from the literature. Table1 shows the identified features of crowd based on literature.


### 3.1. Experimental Setup

The process of implementation and experimentation was done in the Superdecision software. The features were given as input to the software and then plotted as a hierarchy of goals, criteria, and alternatives. Figure2 shows the process of making a hierarchy of the features along with the alternatives of crowds with the goal of selecting the crowd from the available options.Figure 2
Hierarchy of features and crowds for selection of crowd.After plotting the features and crowd, the process of comparison was then done for each feature with respect to each crowd. For the information here only one comparison is shown. The same process is done for the comparison of all features and all crowds. Figure3 graphically represents the process of comparison.Figure 3
Proposed process of comparison.The values were given to each feature and then crowd. This process was done through the support of the tool. Figure4 represents the graphical representation of the weights to each feature.Figure 4
Assignment of weights to the feature.After assigning relevant weights to each feature and crowd, the process of comparison was done and the unweighted, weighted, and limit matrices were obtained for making the selection decision of crowd.


## 3.1. Experimental Setup

The process of implementation and experimentation was done in the Superdecision software. The features were given as input to the software and then plotted as a hierarchy of goals, criteria, and alternatives. Figure2 shows the process of making a hierarchy of the features along with the alternatives of crowds with the goal of selecting the crowd from the available options.Figure 2
Hierarchy of features and crowds for selection of crowd.After plotting the features and crowd, the process of comparison was then done for each feature with respect to each crowd. For the information here only one comparison is shown. The same process is done for the comparison of all features and all crowds. Figure3 graphically represents the process of comparison.Figure 3
Proposed process of comparison.The values were given to each feature and then crowd. This process was done through the support of the tool. Figure4 represents the graphical representation of the weights to each feature.Figure 4
Assignment of weights to the feature.After assigning relevant weights to each feature and crowd, the process of comparison was done and the unweighted, weighted, and limit matrices were obtained for making the selection decision of crowd.


## 4. Results and Discussion

Crowdsourcing is a complex task-solving model to utilize the efforts of humans for solving organizational-specific issues. For assigning a crowdsourced task to an online crowd, the process of selecting a crowd is carried out to select a suitable crowd for attaining the given task. Making an early decision associated with the selection of the crowd can ultimately lead to successful completion of tasks. For selecting the best and right crowd from the crowdsourcing, this research presents a DSS for the appropriate selection of the crowd. The approach has been implemented in the Superdecision tool by plotting the hierarchy of goals, criteria, and alternatives. Various comparisons of the identified features with respect to crowds and crowds with respect to each feature were done. Relevant weights were given and after completion of the comparison process, different results were obtained. These results are shown in the form of tables and figures. Figure5 graphically represents the priorities of features and available crowds based on normalization process by cluster and limiting.Figure 5
Normalization of priorities based on cluster and limiting.As shown in Section3, all the process of pairwise comparisons has been done in the software and for understanding only one representation is given as shown in Figures 3 and 4. The same processes have been done in the software for the rest of the attributes and alternatives. Once, all the comparisons process completed then all the normalized values of each criteria and alternatives are brought into unweighted and weighted super matrix. In unweighted matrix, the sum column values are greater than 1; then it is normalized again and then converted into weighted super matrix.After the pairwise comparisons, all resulting comparisons for features and crowds were integrated into an unweighted matrix. The unweighted matrix is the collection of all pairwise comparisons done in the proposed research. Table2 shows the unweighted matrix.Table 2
Unweighted matrix.
Crowd1Crowd2Crowd3Crowd4Crowd5Collabo∼Compete∼Decisio∼Experie∼Problem∼Profess∼Qualifi∼SkillTrustwo∼Crowds∼Crowd10.000000.000000.000000.000000.000000.124190.257890.229560.231550.247110.152900.266530.229350.329280.00000Crowd20.000000.000000.000000.000000.000000.451310.151170.369340.392270.351380.373280.313600.399510.252570.00000Crowd30.000000.000000.000000.000000.000000.224340.157470.185340.170840.165920.238900.170190.127120.207740.00000Crowd40.000000.000000.000000.000000.000000.137830.096750.083520.083950.102140.101890.097300.077570.120230.00000Crowd50.000000.000000.000000.000000.000000.062340.336710.132240.121390.133440.133030.152380.166450.090170.00000Collabo∼0.163950.217120.186550.104240.088960.000000.000000.000000.000000.000000.000000.000000.000000.000000.20414Compete∼0.214790.172160.207170.215850.041650.000000.000000.000000.000000.000000.000000.000000.000000.000000.11819Decisio∼0.150870.111610.136020.172320.164900.000000.000000.000000.000000.000000.000000.000000.000000.000000.15687Experie∼0.099660.133800.090270.149650.204180.000000.000000.000000.000000.000000.000000.000000.000000.000000.04996Problem∼0.076130.071700.105020.089310.121310.000000.000000.000000.000000.000000.000000.000000.000000.000000.09886Profess∼0.097560.098740.101670.090430.119900.000000.000000.000000.000000.000000.000000.000000.000000.000000.13698Qualifi∼0.069000.052360.065380.065880.090600.000000.000000.000000.000000.000000.000000.000000.000000.000000.09931Skill0.075520.039980.058990.051640.097310.000000.000000.000000.000000.000000.000000.000000.000000.000000.06751Trustwo∼0.052510.102520.048940.060670.071190.000000.000000.000000.000000.000000.000000.000000.000000.000000.06817Crowds∼0.000000.000000.000000.000000.000000.000000.000000.000000.000000.000000.000000.000000.000000.000000.00000The unweighted matrix was then normalized for obtaining the weighted matrix. Table3 represents the weighted matrix.Table 3
Weighted matrix.
Crowd1Crowd2Crowd3Crowd4Crowd5Collabo∼Compete∼Decisio∼Experie∼Problem∼Profess∼Qualifi∼SkillTrustwo∼Crowds∼Crowd10.000000.000000.000000.000000.000000.124190.257890.229560.231550.247110.152900.266530.229350.329280.00000Crowd20.000000.000000.000000.000000.000000.451310.151170.369340.392270.351380.373280.313600.399510.252570.00000Crowd30.000000.000000.000000.000000.000000.224340.157470.185340.170840.165920.238900.170190.127120.207740.00000Crowd40.000000.000000.000000.000000.000000.137830.096750.083520.083950.102140.101890.097300.077570.120230.00000Crowd50.000000.000000.000000.000000.000000.062340.336710.132240.121390.133440.133030.152380.166450.090170.00000Collabo∼0.163950.217120.186550.104240.088960.000000.000000.000000.000000.000000.000000.000000.000000.000000.20414Compete∼0.214790.172160.207170.215850.041650.000000.000000.000000.000000.000000.000000.000000.000000.000000.11819Decisio∼0.150870.111610.136020.172320.164900.000000.000000.000000.000000.000000.000000.000000.000000.000000.15687Experie∼0.099660.133800.090270.149650.204180.000000.000000.000000.000000.000000.000000.000000.000000.000000.04996Problem∼0.076130.071700.105020.089310.121310.000000.000000.000000.000000.000000.000000.000000.000000.000000.09886Profess∼0.097560.098740.101670.090430.119900.000000.000000.000000.000000.000000.000000.000000.000000.000000.13698Qualifi∼0.069000.052360.065380.065880.090600.000000.000000.000000.000000.000000.000000.000000.000000.000000.09931Skill0.075520.039980.058990.051640.097310.000000.000000.000000.000000.000000.000000.000000.000000.000000.06751Trustwo∼0.052510.102520.048940.060670.071190.000000.000000.000000.000000.000000.000000.000000.000000.000000.06817Crowds∼0.000000.000000.000000.000000.000000.000000.000000.000000.000000.000000.000000.000000.000000.000000.00000The weighted matrix was then converted to the limit matrix which is the final matrix for making the decision. The limit matrix was obtained by taking the power of the weighted matrix. Table4 represents the limit matrix. From this matrix, the decision regarding the crowd can be made.Table 4
Limit matrix.
Crowd1Crowd2Crowd3Crowd4Crowd5Collabo∼Compete∼Decisio∼Experie∼Problem∼Profess∼Qualifi∼SkillTrustwo∼Crowds∼Crowd10.110190.110190.110190.110190.110190.110190.110190.110190.110190.110190.110190.110190.110190.110190.11019Crowd20.168550.168550.168550.168550.168550.168550.168550.168550.168550.168550.168550.168550.168550.168550.16855Crowd30.092990.092990.092990.092990.092990.092990.092990.092990.092990.092990.092990.092990.092990.092990.09299Crowd40.050850.050850.050850.050850.050850.050850.050850.050850.050850.050850.050850.050850.050850.050850.05085Crowd50.077420.077420.077420.077420.077420.077420.077420.077420.077420.077420.077420.077420.077420.077420.07742Collabo∼0.084200.084200.084200.084200.084200.084200.084200.084200.084200.084200.084200.084200.084200.084200.08420Compete∼0.086150.086150.086150.086150.086150.086150.086150.086150.086150.086150.086150.086150.086150.086150.08615Decision∼0.069610.069610.069610.069610.069610.069610.069610.069610.069610.069610.069610.069610.069610.069610.06961Experie∼0.065350.065350.065350.065350.065350.065350.065350.065350.065350.065350.065350.065350.065350.065350.06535Problem∼0.044170.044170.044170.044170.044170.044170.044170.044170.044170.044170.044170.044170.044170.044170.04417Profess∼0.050730.050730.050730.050730.050730.050730.050730.050730.050730.050730.050730.050730.050730.050730.05073Qualifi∼0.032870.032870.032870.032870.032870.032870.032870.032870.032870.032870.032870.032870.032870.032870.03287Skill0.030710.030710.030710.030710.030710.030710.030710.030710.030710.030710.030710.030710.030710.030710.03071Trustwo∼0.036210.036210.036210.036210.036210.036210.036210.036210.036210.036210.036210.036210.036210.036210.03621Crowds∼0.000000.000000.000000.000000.000000.000000.000000.000000.000000.000000.000000.000000.000000.000000.00000Figure6 graphically shows the ranking of available crowds. Among the available alternatives of crowds, crowd2 has obtained the highest score which was considered as the highest priority, followed by crowd1, and so on. Therefore, from this figure, one can make decisions regarding the selection of the best crowd among the available alternatives.Figure 6
Ranking of crowds.


## 5. Conclusion

Crowds are online people who have the capabilities to complete diverse types of tasks and projects. These crowds may be new comers who are accomplishing tasks for first time or they may be experienced members who have already finished various tasks in preceding projects. Crowdsourcing is a composite task-solving approach utilizing humans for solving organizational explicit problems. For assessing the crowdsourced task with online crowd, the crowd selection is carried out for the selection of an optimal and appropriate crowd for achieving the task. Early and on-time decision associated with the selection of the crowd can eventually put forward the successful completion of tasks. To select the most appropriate crowd from the crowdsourcing, the present study endeavors to attempt and devise a DSS for the selection of crowd from the crowdsource. The proposed DSS has been executed in the Superdecision tool. In the given tool, the hierarchy of criteria, alternatives, and goal was defined and then a process of pairwise comparisons has been done. Each table of pairwise comparison process was normalized in order to achieve optimal results for the selection of appropriate crowd. The experimental results of the study show that the proposed DSS is efficient and effective for the appropriate selection of crowd in crowdsourcing. In the future, the applicability of the proposed DSS will be tried through various parameters against robustness of the system and its effectiveness will be checked for effective usage in the crowdsource projects.



---

*Source: 5518878-2021-06-10.xml*