# Enabling Flexible IT Services by Crowdsourcing: A Method for Estimating Crowdsourcing Participants

Yiwei Gong

► To cite this version:

Yiwei Gong. Enabling Flexible IT Services by Crowdsourcing: A Method for Estimating Crowdsourcing Participants. 14th Conference on e-Business, e-Services and e-Society (I3E), Oct 2015, Delft, Netherlands. pp.275-286, 10.1007/978-3-319-25013-7_22. hal-01448046

HAL Id: hal-01448046

https://inria.hal.science/hal-01448046

Submitted on 27 Jan 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Enabling Flexible IT Services by Crowdsourcing: A
Method for Estimating Crowdsourcing Participants

Yiwei Gong

School of Information Management, Wuhan University,
Wuhan, Hubei, 430072 P.R. China
yiweigong@whu.edu.cn

**Abstract.** Crowdsourcing has become an increasingly attractive practice for companies to execute business processes in open contexts with on-demand workforce and higher level of flexibility. One of the challenges is the identification of the best-fit crowdsourcing participant from a group of online candidates. This paper presents a method of AHP-TOPSIS based on Grey Relation Analysis for estimating participants of a crowdsourcing task based on their online profiles and proposals. This method is tested by an experiment on a dataset of 348 completed IT service crowdsourcing tasks. An analysis on the matching between the test result and the actual selection result reveals the accuracy and efficiency of this method. Companies can use this method to facilitate the quality control at the beginning of crowdsourcing and keeps the selection of participants easy. This paper contributes to the design of a software agent for crowdsourcing platforms to automatically rank the participants of a task.

**Keywords:** Crowdsourcing, Flexibility, AHP, TOPSIS, Grey Relation Analysis

# 1 Introduction

The concept of crowdsourcing is first coined in 2006 and simply means outsourcing certain tasks and problem formulations to an undefined (and generally large) network of people in the form of open calls [1]. With today's development of Internet and mobile technologies, and the explosion of social media, companies are able to have a better engagement of distributed crowds of individuals for their innovation and problem-solving needs [2]. As a result, an increasing number of companies, ranging from small startups to those listed in Fortune 500, are trying to make use of crowdsourcing to access knowledge and skills that previously unavailable to them and to solve parts of business processes formerly executed in-house [3]. In this way, companies can have a more flexible workforce and higher knowledge absorptive capacity and business processes can be adapted on-demand, which results in higher level of process flexibility [4].

Crowdsourcing can be considered as an online and distributed problem-solving model [5] and suggests that engaging crowds can help companies develop solutions to a variety of business challenges. As the business challenges and tasks vary, so do the
knowledge and skills that crowdsourcing participants have. Unlike simple and low-
priced tasks that commonly require general skills, IT service tasks are knowledge-
intensive and require crowdsourcing participants with special skills and knowledge.
This makes the identification of suitable participants a challenge. Matching skills to
tasks often relays on sophisticated online crowdsourcing platforms to manage distrib-
uted workers and support task providers [2]. However, prior research on crowdsourc-
ing focuses on its business models [6, 7] or brand-related and marketing alike activi-
ties [8, 9], taken the online crowdsourcing platforms and their functionalities as given
[10]. The state of the art in crowdsourcing practice still lacks approaches for automat-
ed estimating participants considering their skills and knowledge [11].

In this paper, we propose a method of AHP-TOPSIS based on Grey Relation Anal-
ysis to help companies estimate and identify the best-fit participant for their IT service
crowdsourcing tasks. The method is tested on a dataset of 348 completed crowdsourc-
ing tasks in the IT service domain. A post-hoc analysis on the matching between the
estimation result and the actual decision made by task providers reveals the accuracy
and efficiency of this method.

2 Research Context

2.1 Forms of Crowdsourcing

The way of using crowdsourcing to abstain flexible workforce for business process
execution is similar to cloud computing where computing capacity is provided on
demand [11, 12]. A typical form of crowdsourcing is publishing the request for pro-
posals through an online marketplace platform with the details of the needed service
and its expected duration and (a range of) cost. Then potential participants bid on the
task by submitting their proposals. Although many proposals would be received for a
task, only the best-fit candidate will be selected to carry out the task. At the end, com-
pany can decide to accept and pay for the work, or refuse it if it does not fulfil the
expectation. This marketplace form of crowdsourcing enables companies to access the
vast potential of workforce with various backgrounds, while it has more flexibility
and less risk than having a fixed outsourcing contract [13].

There are also other forms of crowdsourcing such as knowledge contributions (e.g. Wikipedia), rating (i.e. participants 'vote' on a given topic) and micro-task (e.g. Galaxy Zoo) where participants complete the task voluntarily. In addition, contest-based crowdsourcing is used for obtaining innovative ideas or solutions, in which all the participants make their effort and results are determined on a comparative basis and probably only the top contributor(s) would be reworded. Those forms of crowdsourcing are out of the scope of this paper, as they are less effective in providing business process flexibility.

## 2.2 Related Work

Estimation issue in crowdsourcing has been observed by researchers, and there are
some studies for automatically estimating different submissions to find out those with
sufficient quality for a task. For example, Tarasov et al. [14] proposed a dynamic estimation of worker reliability in rating-based crowdsourcing. This approach is for detecting noisy and incompetent workers by estimating their submissions, instead of estimating the workers before the task was taken into execution.

Mechanisms for estimating crowdsourcing participants for business process execu-
tion can be found in the research of BPEL4People in social networks [15], in which a
ranking method based on Hyperlink-Induced Topic Search (HITS) algorithm is pro-
vided to estimate the expertise of works in a social network. In this method, a certain
skill, its expected level and the importance of a task are used as input, and the ranking
result presents a list of all the suitable crowdsourcing works in a social network. The
underlying concept of BPEL4People is that the flexibility of traditional SOA-based
business process systems can be enhanced by enabling human-based services with
very the same API used by software-based Web services. In this way, tasks would be
able to match to suitable workers that are registered and active on the crowdsourcing
social platform [11].

However, not all crowdsourcing platforms take the form of social networks where
crowdsourcing participants work with each other in a joint task and some of them
would take the role of supervisor or coordinator. Instead, many crowdsourcing plat-
forms have a form of marketplaces, where a task is completed by only one participant
exclusively. In this case, many candidates will compete for the same task by submit-
ting their proposals and the task provider has to choose the best-fit one from them.
The more candidates a task has, the more difficult for the task provider to manually
estimate and identify the best-fit participant, because of information overload. It is
therefore important to have an estimation method to rank all the candidates for the
task provider to choose. And a reliable estimation should be a necessary functionality
of online marketplace crowdsourcing platforms to support task providers for quality
control and solving the problem of information overload.

## 2.3 AHP, TOPSIS and GRA

Selecting the best-fit crowdsourcing participant from a group of candidates is a
typical Multiple Attribute Decision Making (MADM) problem [16] in which deci-
sion-making is for selecting the most appropriate one from many feasible solutions.
Analytical Hierarchical Process (AHP) [17] is one of the most outstanding MADM
methods, which first estimates the relationship among criteria weight and then the
total value of each choice based on the obtained weight [18]. The Technique for Order
of Preference by Similarity to Ideal Solution (TOPSIS) is another outstanding MADM
method, which is based on the concept that the best choice should have the shortest
Euclidean distances from the positive ideal and the farthest from the negative ideal
[16]. AHP and TOPSIS can be used in combination [19] where AHP is used to calcu-
late the weights of the parameters and these weights are later used in TOPSIS.

The drawback of TOPSIS method is its linear variation of each alternatives, which
cannot provide an accurate ranking between two alternatives that have the same dis-
tances to the ideal. This problem can be solved by Grey Relational Analysis (GRA)
which is an effective method to solve decision making problems by generalizing es-
timates under limited samples and uncertain conditions [20]. GRA is a kind of flexible measurement of curve similarity. By using GRA, the nonlinear relationship between the sequences of each alternative can be well reflected, which can compensate the inaccuracy problem of TOPSIS method.

AHP-TOPSIS based on GRA has been proved to be useful in solving MADM problems [21]. In this study, it is employed for estimating crowdsourcing participants for given tasks.

# 3 Estimation Method

In this section a method of AHP-TOPSIS based on GRA is proposed for estimating crowdsourcing participants. This method has the following three phases.

## 3.1 Phase 1: Identifying Estimation Parameters

The estimation parameters used by an algorithm-based method should be quantitative, otherwise they cannot be calculated. In addition, the data of parameters should be easy to access, otherwise the desired automation in the ranking of participants cannot be achieved. In this study, there are two underlying assumptions. The first one is that task providers and candidate participants do not know each other in actual life. This means that a task provider makes its selection decision only based on the related candidate participants' information that is available online. The second one is that task providers will insist on looking for the best-fit participant rather than shifting to other strategy like choosing the first acceptable candidate. This means that all related candidate participants should be involved in the consideration during the decision-making. In a typical marketplace crowdsourcing model, participants' online information comes from either their online profiles or the proposals that they submitted to the task. Both these two sources of information are involved in the formulation of the estimation parameters in this study. Afterwards, the parameters that cannot be quantified has to be ignored, and the parameters that reflect the same property are merged. At the end, parameters are categorized into benefit parameters (the larger the value is, the better the solution is) and cost parameters (the smaller the value is, the better the solution is).

## 3.2 Phase 2: Using AHP to calculate the weight of parameters

In AHP, the multi-attribute weight measurement is calculated via pair-wise comparison of the relative importance of two factors. Assuming that there are N number of decision parameters, denoted as ($P_1, P_2, \cdots, P_n$), its judgment matrix would be $A = [a_n]$, in which $a_n$ represents the relative importance of $P_1$ and $P_2$. Using the row vector average normalization proposed by Satty [17], the weight of $P_i$ is calculated as:
$$W_i = \frac{\left( \prod_{j=1}^{n} a_{ij} \right)^{\frac{1}{n}}}{\sum_{i=1}^{n} \left( \prod_{j=1}^{n} a_{ij} \right)^{\frac{1}{n}}} \quad i, j = 1, 2, \dots, n$$

### 3.3 Phase 3: Using GRA-based TOPSIS to Estimate Participants

In this phase the algorithm has the following ten steps.

1. Normalizing of Initial decision matrix $X = (x_{ij})_{m \times n}$, get the normalization matrix $Z = (z_{ij})_{m \times n}$. ($i = 1, 2, \dots, m$; $j = 1, 2, \dots, n$)

For benefit parameters:

$$(1) \quad Z_{ij} = \frac{x_{ij} - \min_{i} x_{ij}}{\max_{i} x_{ij} - \min_{i} x_{ij}}$$

For cost parameters:

$$(2) \quad Z_{ij} = \frac{\max_{i} x_{ij} - x_{ij}}{\max_{i} x_{ij} - \min_{i} x_{ij}}$$

2. Calculating the weighted decision matrix

$$S = (s_{ij})_{m \times n}, (i = 1, 2, \dots, m; j = 1, 2, \dots, n).$$

$$S_{ij} = w_{ij} z_{ij}$$

3. Calculating the positive ideal solution $S^+$ and negative ideal solution $S^-$

$$S^+ = (s_1^+, s_2^+, \dots, s_n^+); \quad S^- = (s_1^-, s_2^-, \dots, s_n^-)$$

Where $s_j^+ = \max_i s_{ij} = w_j$; $s_j^- = \min_i s_{ij} = 0$; $i = 1, 2, \dots, m$; $j = 1, 2, \dots, n$.

4. Calculating the Euclidean distance between each solution and positive/negative ideal solution $d_i^+$, $d_i^-$

$$d_i^+ = \sqrt{\sum_{j=1}^{n} (s_{ij} - s_j^+)^2}, \quad d_i^- = \sqrt{\sum_{j=1}^{n} (s_{ij} - s_j^-)^2}$$
where $i=1,2,\cdots,m; j=1,2,\cdots,n$.

5. Calculating the grey relation coefficient matrix of each solution and positive/negative ideal solution $L^+$, $L^-$:

$$L^+ = (l_{ij}^+)_{m \times n}, \quad L^- = (l_{ij}^-)_{m \times n}$$

$$\text{Where } l_{ij}^{+} = \frac{\min_{i} \min_{j} |s_{j}^{+} - s_{ij}| + \theta \max_{i} \max_{j} |s_{j}^{+} - s_{ij}|}{|s_{j}^{+} - s_{ij}| + \theta \max_{i} \max_{j} |s_{j}^{+} - s_{ij}|}$$

$$l_{ij}^{-} = \frac{\min_{i} \min_{j} |s_{j}^{-} - s_{ij}| + \theta \max_{i} \max_{j} |s_{j}^{-} - s_{ij}|}{|s_{j}^{-} - s_{ij}| + \theta \max_{i} \max_{j} |s_{j}^{-} - s_{ij}|}$$

Where $\theta \in (0,1)$, is distinguishing coefficient. Here the value of $\theta$ is set to be 0.5.

$$\text{Simplifying the formulas: } L_{ij}^{+} = \frac{\theta}{|z_{ij} - 1| + \theta}; \quad L_{ij}^{-} = \frac{\theta}{|z_{ij} + 1| + \theta}$$

6. Calculating the grey relation grade of each solution and positive/negative ideal solution $l_i^+$, $l_i^-$:

$$l_i^+ = \frac{1}{n} \sum_{j=1}^{n} l_{ij}^+; \quad l_i^- = \frac{1}{n} \sum_{j=1}^{n} l_{ij}^-$$

7. Applying nondimensionalization to $d_i^+, d_i^-, l_i^+$ and $l_i^-$:

$$D_i^+ = \frac{d_i^+}{\max_{i} d_i^+}, \quad D_i^- = \frac{d_i^-}{\max_{i} d_i^-}$$

$$L_i^+ = \frac{l_i^+}{\max_{i} l_i^+}, \quad L_i^- = \frac{l_i^-}{\max_{i} l_i^-}$$

Where $i=1,2,\cdots,m$

8. Calculating the relative closeness degree:

$$P_i^+ = \frac{D_i^+}{D_i^+ + D_i^-}, \quad U_i^+ = \frac{L_i^+}{L_i^+ + L_i^-}$$
9. Combining $P_i^+$ and $U_i^+$: $Q_i^+ = v_1 P_i^+ + v_2 U_i^+$

Where $v_1$ and $v_2$ reflect the degree of preference of decision makers, $v_1 + v_2 = 1$, and $v_1 = v_2 = 0.5$

10. Sorting solutions by the value of $Q_i^+$. The better $Q_i^+$ is, the better the solution is, and vice versa. $\max(Q_i^+)$ is the final decision.

# 4 Experiment

In order to evaluate the proposed method of AHP-TOPSIS based on GRA on its accuracy and efficiency in estimating crowdsourcing participants, an experiment was carried out on data from a popular Chinese crowdsourcing marketplace platform, Epweike (http://www.epweike.com/). This experiment used the proposed method to estimate crowdsourcing participants of certain tasks, and then the estimation result was compared with the actual selection decision made by the task provider. This comparison allows for an analysis on the accuracy of the method and the impact of the number of candidates on the actual decision-making which reflects the efficiency of this method.

## 4.1 Dataset

In this experiment, a dataset of 348 valid and completed tasks between 2010 and 2015 for IT services crowdsourcing is used. The content of those tasks includes software/mobile application development, website construction, database and system design, server maintenance, etc. In those tasks the number of openly visible proposals is more than 3. Tasks that have less than 4 proposals are ignored, because the GRA-based TOPSIS algorithm does not return a meaningful ranking result when the number of candidates is less than 4.

## 4.2 Approach

### Phase 1: Identifying Estimation Parameters

The following 10 parameters of crowdsourcing participants are identified for the estimation, and the data of these parameters can be accessed from the website openly.

<table><caption>**Table 1.** The description of each parameter</caption><thead><tr><td>Parameters</td><td>Description</td></tr></thead><tbody><tr><td>P1</td><td>The total volume of the participant in the history</td></tr><tr><td>P2</td><td>The praise rate accumulates from the assessments made by task providers for each task completed by the participant</td></tr></tbody></table>
<table><tr><td>P3</td><td>The number of biddings that the participant participates</td></tr><tr><td>P4</td><td>The degree of matching between the required skills for the task and the skills that the participant has</td></tr><tr><td>P5</td><td>The website's evaluation of the participant on its intelligence, authenticity, trusted transactions and public praise</td></tr><tr><td>P6</td><td>The website's overall evaluation of the participant</td></tr><tr><td>P7</td><td>The price proposed by the participant</td></tr><tr><td>P8</td><td>The task duration proposed by the participant</td></tr><tr><td>P9</td><td>The number of visitors of participant's homepage</td></tr><tr><td>P10</td><td>VIP level</td></tr></table>

These parameters describe the information of either the participant itself or the proposal it provided. Among those parameters, P1, P2, P3, P4, P5, P9 and P10 are benefit parameters; while P6, P7 and P8 are cost parameters.

## Phase 2: Using AHP to Calculate the Weight of Parameters

By AHP, the weight of the 10 parameters is calculated:

$$w_i = (0.14091127, 0.19491718, 0.05137758, 0.04104782, 0.03849227, 0.03083646, 0.25585735, 0.20933824, 0.01622340, 0.02099905)$$

## Phase 3: Using GRA-based TOPSIS to Estimate Participants

For space reason, only the calculation of Task1 is given as an example. This task has 23 candidate participants and 18 openly available proposals. The other 5 proposals are closed and only visible to the task provider, and therefore those 5 participants are not taken into account in the experiment. The best one is identified from the rest 18 candidates by using the GRA-based TOPSIS method. Through the estimation, the final value of $Q_i^+$ is presented in the following table.

<table><caption>**Table 2.** value of $Q_i^+$ for each participant of Task 1</caption><thead><tr><th>Participants</th><th>$Q_i^+$</th><th>Participants</th><th>$Q_i^+$</th></tr></thead><tbody><tr><td>1</td><td>0.481154071</td><td>10</td><td>0.459536143</td></tr><tr><td>2</td><td>0.486380377</td><td>11</td><td>0.481035623</td></tr><tr><td>3</td><td>0.425769924</td><td>12</td><td>0.503450868</td></tr><tr><td>4</td><td>0.512654617</td><td>13</td><td>0.499094489</td></tr><tr><td>5</td><td>0.488922997</td><td>14</td><td>0.619961894</td></tr><tr><td>6</td><td>0.509868043</td><td>15</td><td>0.505601337</td></tr><tr><td>7</td><td>0.479417731</td><td>16</td><td>0.434555178</td></tr></tbody></table>
<table><tr><td>8</td><td>0.51068885</td><td>17</td><td>0.603533483</td></tr><tr><td>9</td><td>0.441772416</td><td>18</td><td>0.479186991</td></tr></table>

By descending the order of $Q_i^+$, the sort of participants of Task 1 can be get.

<table><caption>**Table 3.** The ranking of participants of Task 1</caption><thead><tr><th>Order</th><th>Participants</th><th>Order</th><th>Participants</th></tr></thead><tbody><tr><td>1</td><td>20</td><td>10</td><td>2</td></tr><tr><td>2</td><td>23</td><td>11</td><td>1</td></tr><tr><td>3</td><td>4</td><td>12</td><td>15</td></tr><tr><td>4</td><td>9</td><td>13</td><td>24</td></tr><tr><td>5</td><td>7</td><td>14</td><td>8</td></tr><tr><td>6</td><td>21</td><td>15</td><td>14</td></tr><tr><td>7</td><td>16</td><td>16</td><td>12</td></tr><tr><td>8</td><td>17</td><td>17</td><td>22</td></tr><tr><td>9</td><td>6</td><td>18</td><td>3</td></tr></tbody></table>

According to the result presented in Table 3, the best candidate for Task 1 is Participant 20. The values of each of its parameters are:

$$P_{20} = (17000, 100\%, 4, 4, 40, 5, 800, 5, 12624, 1)$$

But the subjective choice made by the task provider is Participant 1, and the values of its parameters are:

$$P_1 = (9200, 100\%, 4, 3, 39, 6, 500, 2, 2588, 1)$$

In this example, the estimation result is not matching with the actual selection.

In this experiment, Phase 3 was repeated on all the 348 tasks. Then the result of the estimation was compared with the actual selection made by the task provider to find out whether they are matchable. The latter is openly available on the crowdsourcing website.

## 4.3 Result Analysis

In order to evaluate the accuracy of the proposed method, the result of estimation is compared with the actual selection decision made by task providers. A metric of matching rate is used to indicate the percentages of matchable tasks. In this experiment, the overall matching rate is 88.22% (307 out of 348 tasks). The matching rates under different number of participants are presented in the following table.
<table><caption>**Table 4.** Matching rates under different number of participants</caption><thead><tr><th>Number of Participants</th><th>Number of Tasks</th><th>Number of Matched tasks</th><th>Matching Rate</th><th>Number of Participants</th><th>Number of Tasks</th><th>Number of Matched tasks</th><th>Matching Rate</th></tr></thead><tbody><tr><td>4</td><td>65</td><td>65</td><td>100%</td><td>31</td><td>11</td><td>8</td><td>72.73%</td></tr><tr><td>5</td><td>55</td><td>54</td><td>98.18%</td><td>32</td><td>11</td><td>8</td><td>72.73%</td></tr><tr><td>6</td><td>24</td><td>23</td><td>95.83%</td><td>37</td><td>8</td><td>5</td><td>62.5%</td></tr><tr><td>7</td><td>18</td><td>17</td><td>94.44%</td><td>38</td><td>3</td><td>3</td><td>100%</td></tr><tr><td>8</td><td>61</td><td>57</td><td>93.44%</td><td>39</td><td>5</td><td>3</td><td>60%</td></tr><tr><td>9</td><td>15</td><td>14</td><td>93.33%</td><td>40</td><td>4</td><td>2</td><td>50%</td></tr><tr><td>10</td><td>10</td><td>9</td><td>90%</td><td>42</td><td>2</td><td>1</td><td>50%</td></tr><tr><td>11</td><td>5</td><td>5</td><td>100%</td><td>44</td><td>2</td><td>1</td><td>50%</td></tr><tr><td>12</td><td>7</td><td>6</td><td>85.71%</td><td>47</td><td>2</td><td>1</td><td>50%</td></tr><tr><td>15</td><td>4</td><td>4</td><td>100%</td><td>52</td><td>3</td><td>1</td><td>33.33%</td></tr><tr><td>17</td><td>5</td><td>4</td><td>80%</td><td>57</td><td>3</td><td>1</td><td>33.33%</td></tr><tr><td>22</td><td>5</td><td>4</td><td>80%</td><td>60</td><td>1</td><td>0</td><td>0%</td></tr><tr><td>23</td><td>4</td><td>3</td><td>75%</td><td>62</td><td>3</td><td>1</td><td>33.33%</td></tr><tr><td>29</td><td>8</td><td>6</td><td>75%</td><td>63</td><td>4</td><td>1</td><td>25%</td></tr></tbody></table>

The matching rates presented in Table 4 indicate that the proposed method has a high accuracy when the number of participants is between 4 and 10. There are 248 tasks (71.26% of the total 348 tasks) fall into this range and the matching rates are above 90%. Specially, in the simplest situation where only 4 participants competing a task, the estimation result is 100% matched with the manual decision-making. This means the proposed method can achieve an estimation result that is very similar with the manual decision-making, when the manual decision-making is simple and the information overload problem does not appear.

Technically speaking, the accuracy of this method will not be influenced by the number of participants. However, the matching rates generally declines along with the increase of participants. A possible explanation is that when the number of participants increase, it is more difficult to manually identify the best-fit participant. The task provider might have to rely on its own experience or even instinct in the decision-making for participant selection rather than an objective comparison between candidates. When the number of participants is more than 50, manually identifying the best-fit participant becomes very difficult and the actual selection result deviates very much from the estimation result, which results in matching rates below 33.33%. The following curve demonstrate the decline of the matching rate. Under the assumption that the accuracy of manual decision-making is mainly and negatively influenced by information overload, the proposed method could solve this problem and improve the decision-making for crowdsourcing tasks with a large number of participants. To
solidly prove this statement, investigation on the factors that impact manual decision-making for selecting crowdsourcing participants is desired in the future.

<figure><img src="image_2.png" /><figcaption>Fig. 1. The matching rates decline along with the increase of participants</figcaption></figure>

# 5 Conclusions and Future Work

Companies want to adopt crowdsourcing to obtain on-demand IT services and enable flexible business processes. The challenge to overcome the information overload problem in estimating and identifying the best-fit crowdsourcing participant out of a large group of candidates. This paper presents a method of AHP-TOPSIS based on GRA for estimating crowdsourcing participants. The proposed method has been tested on a dataset of 348 valid and completed IT service crowdsourcing tasks from a Chinese crowdsourcing online marketplace platform. For the tasks with a small number of candidates, a matching between the estimation result and the actual selection made by task providers proofs the accuracy of the proposed method. Although the matching rate generally declines along with the increase of the number of candidates, this reflects the information overload problem in manual decision-making. The proposed method could solve this problem by providing an accurate and objective estimation which is uninfluenced by the number of participants. Employing this method allows task providers to quickly and easily identify the best-fit crowdsourcing participant. Furthermore, this method facilitates the design of a software agent for crowdsourcing platforms to rank the participants of a task automatically.

In the future work, it would be interesting to contact the task providers who had to make decision with a large number of candidates, especially those chose a participant other than the one recommended by the proposed method. Then invite them to interviews or surveys with questions such as how long it took them to make a decision on selecting the best-fit participant, what are the factors impacting their decision, and whether they think this method would help them in overcoming the information overload problem.
References

1. Howe, J.: The Rise of Crowdsourcing. Wired Magazine, vol. 14. Condé Nast (2006)

2. Lakhani, K.R.: Using the Crowd as an Innovation Partner. Harvard Business Review 91, 60-69 (2013)

3. Afuah, A., Tucci, C.L.: Crowdsourcing As a Solution to Distant Search. Academy of Management Review 37, 355-375 (2012)

4. Gong, Y., Janssen, M.: From policy implementation to business process management: Principles for creating flexibility and agility. Government Information Quarterly 29, 61-71 (2012)

5. Brabham, D.C.: Crowdsourcing as a Model for Problem Solving. Convergence: The International Journal of Research into New Media Technologies 14, 75-90 (2008)

6. Prpic, J., Shukla, P.P., Kietzmann, J.H., McCarthy, I.P.: How to work a crowd: Developing crowd capital through crowdsourcing. Business Horizons 58, 77-85 (2015)

7. Simula, H., Ahola, T.: A network perspective on idea and innovation crowdsourcing in industrial firms. Industrial Marketing Management 43, 400-408 (2014)

8. Djelassi, S., Decoopman, I.: Customers' participation in product development through crowdsourcing: Issues and implications. Industrial Marketing Management 42, 683-692 (2013)

9. Gatautis, R., Vitkauskaitė, E.: Crowdsourcing Application in Marketing Activities. Procedia - Social and Behavioral Sciences 110, 1243-1250 (2013)

10. Majchrzak, A., Malhotra, A.: Towards an information systems perspective and research agenda on crowdsourcing for innovation. Journal of Strategic Information Systems 22, 257-268 (2013)

11. Satzger, B., Psaier, H., Schall, D., Dustdar, S.: Auction-based crowdsourcing supporting skill management. Information Systems 38, 547-560 (2013)

12. Nevo, D., Kotlarsky, J.: Primary vendor capabilities in a mediated outsourcing model: Can IT service providers leverage crowdsourcing? Decision Support Systems 65, 17-27 (2014)

13. Gefen, D., Gefen, G., Carmel, E.: How project description length and expected duration affect bidding and project success in crowdsourcing software development. Journal of Systems and Software (20015)

14. Tarasov, A., Delany, S.J., Namee, B.M.: Dynamic estimation of worker reliability in crowdsourcing for regression tasks: Making it work. Expert Systems with Applications 41, 6190-6210 (2014)

15. Schall, D., Satzger, B., Psaier, H.: Crowdsourcing tasks to social networks in BPEL4People. World Wide Web 17, 1-32 (2014)

16. Hwang, C.-L., Yoon, K.: Multiple Attribute Decision Making: Methods and Applications A State-of-the-Art Survey. Springer-Verlag (1985)

17. Saaty, T.L.: The Analytic Hierarchy Process: Planning, Priority Setting, Resource Allocation. Mcgraw-Hill (1980)

18. Ngai, E.W.T., Chan, E.W.C.: Evaluation of knowledge management tools using AHP. Expert Systems with Applications 29, 889-899 (2005)

19. Patil, S.K., Kant, R.: A fuzzy AHP-TOPSIS framework for ranking the solutions of Knowledge Management adoption in Supply Chain to overcome its barriers. Expert Systems with Applications 41, 679-693 (2014)

20. Deng, J.: Introduction to Grey System. Journal of Grey System 1, 1-24 (1989)

21. Chen, M.-F.: Combining grey relation and TOPSIS concepts for selecting an expatriate host country. Mathematical and Computer Modelling 40, 1473-1490 (2004)