# Recommendation of Crowdsourcing Tasks Based on Word2vec Semantic Tags

Authors: Qingxian Pan; Hongbin Dong; Yingjie Wang; Zhipeng Cai; Lizong Zhang
Journal: Wireless Communications and Mobile Computing
 (2019)

---

## Abstract

Crowdsourcing is the perfect show of collective intelligence, and the key of finishing perfectly the crowdsourcing task is to allocate the appropriate task to the appropriate worker. Now the most of crowdsourcing platforms select tasks through tasks search, but it is short of individual recommendation of tasks. Tag-semantic task recommendation model based on deep learning is proposed in the paper. In this paper, the similarity of word vectors is computed, and the semantic tags similar matrix database is established based on the Word2vec deep learning. The task recommending model is established based on semantic tags to achieve the individual recommendation of crowdsourcing tasks. Through computing the similarity of tags, the relevance between task and worker is obtained, which improves the robustness of task recommendation. Through conducting comparison experiments on Tianpeng web dataset, the effectiveness and applicability of the proposed model are verified.


---

## Body

## 1. Introduction

Deep learning was proposed by Geoffrey Hinton et al. in 2006. This method simulates human brain neural network to model and realize multiple level abstraction [1, 2]. In 2006, Jeff Howe of American Wired magazine reporter proposed crowdsourcing concept [3]. As a new kind of business model, crowdsourcing has been widespread concern in various fields and becomes the new hot point of computer research fields. Task requester, crowdsourcing platform, and worker make up crowdsourcing system [4]. The process of crowdsourcing includes designing task, publishing task, selecting task, sensing task, submitting solution, and integrating solution. Among them, task selection is the key phase in the process of crowdsourcing. This is the key to complete crowdsourcing task that the appropriate worker selects appropriate task in appropriate time [5].The popular crowdsourcing platforms use task searching to get the favourite task by keyword searching [6]. However, with the rapid development of crowdsourcing, the problem of information overload is more and more serious. In addition, it is more and more difficult to get the favourite crowdsourcing task for worker. Recommender system is an effective medium to solve the problem, which is used on many E-Commerce Platforms, such as Alibaba, Amazon, and Netflix [7]. But there are many problems which are not solved in recommender systems, such as similarity calculation, the lower recommended accuracy, data sparseness, and cold boot. In brief, improving the accuracy and reliability of recommender systems has been paid more attention by scholars.However, individual recommendation research of the task is lesser in crowdsourcing, and task selection is relied on hobbies and expertise. Few crowdsourcing platforms can actively recommend task. This paper researches the crowdsourcing tasks recommendation model based on Word2vec semantic tags in order to achieve individual recommendation of crowdsourcing tasks [8].The main contributions of this paper include following three contents:(1)
Compute the similarity of word vectors and build the semantic tags similar matrix database based on the Word2vec deep learning.(2)
Research the task recommending model based on semantic tags to achieve the individual recommendation of crowdsourcing tasks. This paper computes similarity of tasks and workers based on the semantic tag similar matrix.(3)
Utilizing the Tianpeng Web dataset, the experiments are conducted. The experimental results show that the model is feasible and effective. The model can be used in other fields according to the different semantic databases.This paper is organized as follows. Section2 reviews the related works. The Work2vec is discussed in Section 3. In addition, the tasks recommendation model and realization method based on semantic tags are researched in Section 4. The comparison experiments, as well as the analysis for the experimental results, are introduced in Section 5. The conclusion is presented in Section 6.


## 2. Related Works

In order to discuss the related works for recommendation of crowdsourcing, we, respectively, introduce the related works of crowdsourcing and recommendations.


### 2.1. Crowdsourcing

In 2006, Jeff Howe proposed crowdsourcing concept firstly [3]: a company or an institution outsources the tasks performed by an employee in the past to an unspecific public network in a free and voluntary manner. With the development of crowdsourcing technology, the different crowdsourcing concepts appeared. Chen et al. [9] summarized 40 different crowdsourcing definitions. Feng et al. [10] gave the definition of crowdsourcing according to the basic features of crowdsourcing. According to the definition, crowdsourcing is a distributed problem-solving mechanism opening to the Internet public, and it completes the tasks that are difficult to complete by a computer through integrating computers and the unknown public on the Internet [11].Crowdsourcing is successfully applied in language translation, image recognition, intelligent transportation, software development, entry interpretation, tourism photography, and other fields, which has become the perfect embodiment of group wisdom [12, 13]. Crowdsourcing is made up of the task requester, crowdsourcing platform, and workers. The crowdsourcing workflow includes designing tasks by task requester, publishing tasks, selecting tasks by workers, solving tasks, submitting answer, and arranging answer. The workflow of crowdsourcing is shown by Figure 1. The public participation is the basis of crowdsourcing. And the key to high-quality complete crowdsourcing tasks is to recommend appropriate tasks to appropriate worker in appropriate time [14].Figure 1
The workflow of crowdsourcing.


### 2.2. Recommender Systems

With the arrival of big data era, the problem of information overload is more and more serious and that finding the useful and best information is more and more difficult. Recommender Systems is an effective medium to solve the above problems [15]. However, there are some inherent defects in recommendation systems, such as low accuracy, data sparseness, cold boot, the defects of the centralized system, similarity calculation, and being easy to be attacked. In addition, many recommender systems applied to business systems, whose purpose is to sell more goods and seek the maximum benefits, rather than to recommend the best commodities to users. In brief, the credibility and accuracy of recommendation systems need to be improved, which has attracted the attention of scholars. Yang et al. [16] proposed a recommender system based on transfer learning. Chen et al. [17] proposed a recommender system based on bind context. Tang et al. [18] researched recommender system based on crossing knowledge. Liu [19] and Zhou et al. [20] researched recommender systems for social recommendation. Combining Markov and social attributes of users, Wang et al. [21] proposed a probability-based recommendation model to recommend items for users.Crowdsourcing task recommendation is mainly from the perspective of crowdsourcing platform. Based on the task discovery model, crowdsourcing platform recommends related tasks according to the preferences of workers [5]. The main crowdsourcing platforms basically adopt the way of task search and rarely adopt the method of task recommendation [22]. Some task recommendation methods were researched based on traditional recommendation methods, including content-based recommendation, collaborative filtering, and mixed recommendation algorithms. Ambati et al. [23] proposed the use of task and workers' historical information for task recommendation. Yuen et al. [24] proposed a worker-task recommendation model through combining the historical information of workers and browsing history. Deng et al. [25] researched the problem of maximizing task selection for spatiotemporal tasks.


## 2.1. Crowdsourcing

In 2006, Jeff Howe proposed crowdsourcing concept firstly [3]: a company or an institution outsources the tasks performed by an employee in the past to an unspecific public network in a free and voluntary manner. With the development of crowdsourcing technology, the different crowdsourcing concepts appeared. Chen et al. [9] summarized 40 different crowdsourcing definitions. Feng et al. [10] gave the definition of crowdsourcing according to the basic features of crowdsourcing. According to the definition, crowdsourcing is a distributed problem-solving mechanism opening to the Internet public, and it completes the tasks that are difficult to complete by a computer through integrating computers and the unknown public on the Internet [11].Crowdsourcing is successfully applied in language translation, image recognition, intelligent transportation, software development, entry interpretation, tourism photography, and other fields, which has become the perfect embodiment of group wisdom [12, 13]. Crowdsourcing is made up of the task requester, crowdsourcing platform, and workers. The crowdsourcing workflow includes designing tasks by task requester, publishing tasks, selecting tasks by workers, solving tasks, submitting answer, and arranging answer. The workflow of crowdsourcing is shown by Figure 1. The public participation is the basis of crowdsourcing. And the key to high-quality complete crowdsourcing tasks is to recommend appropriate tasks to appropriate worker in appropriate time [14].Figure 1
The workflow of crowdsourcing.


## 2.2. Recommender Systems

With the arrival of big data era, the problem of information overload is more and more serious and that finding the useful and best information is more and more difficult. Recommender Systems is an effective medium to solve the above problems [15]. However, there are some inherent defects in recommendation systems, such as low accuracy, data sparseness, cold boot, the defects of the centralized system, similarity calculation, and being easy to be attacked. In addition, many recommender systems applied to business systems, whose purpose is to sell more goods and seek the maximum benefits, rather than to recommend the best commodities to users. In brief, the credibility and accuracy of recommendation systems need to be improved, which has attracted the attention of scholars. Yang et al. [16] proposed a recommender system based on transfer learning. Chen et al. [17] proposed a recommender system based on bind context. Tang et al. [18] researched recommender system based on crossing knowledge. Liu [19] and Zhou et al. [20] researched recommender systems for social recommendation. Combining Markov and social attributes of users, Wang et al. [21] proposed a probability-based recommendation model to recommend items for users.Crowdsourcing task recommendation is mainly from the perspective of crowdsourcing platform. Based on the task discovery model, crowdsourcing platform recommends related tasks according to the preferences of workers [5]. The main crowdsourcing platforms basically adopt the way of task search and rarely adopt the method of task recommendation [22]. Some task recommendation methods were researched based on traditional recommendation methods, including content-based recommendation, collaborative filtering, and mixed recommendation algorithms. Ambati et al. [23] proposed the use of task and workers' historical information for task recommendation. Yuen et al. [24] proposed a worker-task recommendation model through combining the historical information of workers and browsing history. Deng et al. [25] researched the problem of maximizing task selection for spatiotemporal tasks.


## 3. Word2vec

In 2003, Bengio et al. [26] proposed Neural Network Language Model-NNLM based on 3 levels. NNLM is used to compute the probability p(wt=i∣context) of the next word wt of a context, and word vector is the byproduct during training. Word2vec is a tool based on deep learning to compute the similarity of word vector which was proposed by Google company in 2013 [27]. It converts the word into word vector and computes similarity according to the cosine between word vectors. When using the tool, the texts after segmentation are input, and the output-word vector can be used to do a lot of Natural Language Processing (NLP) related work, such as clustering, looking for synonyms, and part of speech analysis.Word2vec uses word vector presentation mode based on Distributed representation. Distributed representation is proposed by Hinton in 1986 [28]. Its basic thought is to map each word into a k-dimension real vector by training (k is a hyperparameter in the model) and to judge the semantic similarity between them according to the distance between words (such as cosine similarity, Euclidean distance). It uses a ‘3 layers neural network’, input layer-hidden layer-output layer. Its core technology is to use Huffman code according to word frequency, which makes the activated content basically consistent of all word frequency similar words in hidden layer. The higher the frequency of the word, the less the number of hidden layers they activate, which effectively reduces the computational complexity.Compared with Latent Semantic Index-LSI and Latent Dirichlet Allocation-LDA, Word2vec uses the context of words and makes the semantic information richer. There are two kinds of training model-CBOW (Continuous Bag-of-Words) and Skip-gram in Word2vec, which are shown by Figure2. Two models both include input layer, projection layer, and output layer. CBOW model predicts the current words according to the known context, and Skip-gram model predicts context according to the current words.Figure 2
CBOW model and Skip-gram model.
(a)
CBOW model (b)
Skip-gram modelIn this paper, the objective optimization function of CBOW is expressed by(1)pw∣Contextw=∏j=2lwpdjw∣xw,θj-1wwhere xw means the word vector of the root node in the Hoffman tree, Context(w) represents the context of word w, that is, the collection of w peripheral words, lwrepresents the nodes number of the path pw, and djw∈{0,1} represents Huffman code of the word w; θ1w,θ2w,...,θj-1w∈Rm represents the vectors corresponding to nonleaf nodes of the path pw. Therefore, the logistic regression probability p(djw∣xw,θj-1w) that w passes a node j in the Hoffman tree is shown by (2). The corresponding parameter σ(xwTθj-1w) is shown by (3).(2)pdjw∣xw,θj-1w=σxwTθj-1w,djw=0;1-σxwTθj-1w,djw=1.(3)σxwTθj-1w=11+e-xwTθj-1wIn order to clearly represent the meaning of logistic regression probability p(djwxw,θj-1w), we combine (2) and (3) to obtain the value of p(djwxw,θj-1w), which is shown by (4)pdjw∣xw,θj-1w=σxwTθj-1w1-djw·1-σxwTθj-1wdjwFor avoiding the value of p(w∣Context(w)) too small, logarithm Likelihood function is used to represent the objective function; thus, (1) can be converted into (5)L=∑w∈Clog⁡pw∣ContextwThrough combining (4) and (5), the objective function L is shown by (6)L=∑w∈Clog∏j=2lwσxwTθj-1w1-djw·1-σxwTθj-1wdjw=∑w∈C∑j=2lw1-djw·logσxwTθj-1w+djw·log1-σxwTθj-1wTherefore, (6) is the object function of CBOW in this paper. Word2vec uses random gradient ascent method to optimize the object function of CBOW.


## 4. The Tasks Recommendation Model and Realization Method Based on Semantic Tags

### 4.1. Basic Model Frame and Mathematical Computation Model

The results and discussion may be presented separately, or in one combined section, and may optionally be divided into headed subsections.The core of the model is the research of tag similar matrix. The model uses tag similar matrix to compute the similarity of workers and tasks, produces worker-tag similar matrix, and realizes tasks recommendation or workers recommendation. In model, tag similar matrix is obtained by Word2vec computing. Worker-tag matrix is got according to history work information of the worker, registration information, etc. And task-tag matrix is got according to task description, task classification, etc.Define tag similar matrixL∈Rm×m, l11⋯l1m⋮⋱⋮lm1⋯lmm, L is a symmetric matrix, that is, lij=lji, lij represents the similarity of tag i and tag j, lij∈[0,1], and its value is got through using Word2vec tool to compute. Define worker-tag matrix W∈Rn×m, w11⋯w1m⋮⋱⋮wn1⋯wnm, and, among them, wij=1,workerihastagj;0,workerihasnottagj.We define the task-tag matrixT∈Rp×m, t11⋯t1m⋮⋱⋮tp1⋯tpm, and, among them, tij=1,taskihastagj;0,taskihasnottagj.Therefore, the worker-task similar matrixWT is obtained by (7), where W is the worker-tag matrix, L is the tag similar matrix, and TT means the task-tag transposed matrix. Through (7), the relationship between workers and tasks can be obtained.(7)WT=W×L×TT=w11⋯w1m⋮⋱⋮wn1⋯wnm×l11⋯l1m⋮⋱⋮lm1⋯lmm×t11⋯t1m⋮⋱⋮tp1⋯tpmT


### 4.2. Basic Flow

The main steps of the process of the proposed recommendation model are shown as follows: (1) compute the word vectors based on Word2vec; (2) computing the similarity of word vectors; (3) generating the tag similar matrix; (4) obtaining the worker-tag matrix and task-tag matrix; (5) computing the worker-task similarity matrix; (6)L2 standardization and normalization; (7) tasks and workers recommendation. Tag similar matrix generation uses Word2vec tool. Worker-task similarity computation uses mathematical methods introduced in the previous section. The section mainly introduces standardization and normalization method.L 2 standardization method: the L2 norm definition of vector x(x1,x2,…,xn) is shown as follows: norm(x)=x12+x22+⋯+xn2.In order to makex normalized to the unit L2 norm, the mapping between x and x′ is established, so that the L2 norm of x′ is 1, and the proof is shown as follows:(8)1=normx′=x12+x22+⋯+xn2normx=x1′2+x2′2+⋯+xn′2=x1normx2+x2normx2+⋯+xnnormx2where the value of xi′ is shown by (9)xi′=xinormxIn order to get the standardization and generality of data, the standardization data of L2 is normalized, so that the data fall in the interval [0,1], the conversion formula is shown by (10), where min(X) means the minimum in X, and max(X) is the maximum in X.(10)xi′=xi-min⁡Xmax⁡X-min⁡X


## 4.1. Basic Model Frame and Mathematical Computation Model

The results and discussion may be presented separately, or in one combined section, and may optionally be divided into headed subsections.The core of the model is the research of tag similar matrix. The model uses tag similar matrix to compute the similarity of workers and tasks, produces worker-tag similar matrix, and realizes tasks recommendation or workers recommendation. In model, tag similar matrix is obtained by Word2vec computing. Worker-tag matrix is got according to history work information of the worker, registration information, etc. And task-tag matrix is got according to task description, task classification, etc.Define tag similar matrixL∈Rm×m, l11⋯l1m⋮⋱⋮lm1⋯lmm, L is a symmetric matrix, that is, lij=lji, lij represents the similarity of tag i and tag j, lij∈[0,1], and its value is got through using Word2vec tool to compute. Define worker-tag matrix W∈Rn×m, w11⋯w1m⋮⋱⋮wn1⋯wnm, and, among them, wij=1,workerihastagj;0,workerihasnottagj.We define the task-tag matrixT∈Rp×m, t11⋯t1m⋮⋱⋮tp1⋯tpm, and, among them, tij=1,taskihastagj;0,taskihasnottagj.Therefore, the worker-task similar matrixWT is obtained by (7), where W is the worker-tag matrix, L is the tag similar matrix, and TT means the task-tag transposed matrix. Through (7), the relationship between workers and tasks can be obtained.(7)WT=W×L×TT=w11⋯w1m⋮⋱⋮wn1⋯wnm×l11⋯l1m⋮⋱⋮lm1⋯lmm×t11⋯t1m⋮⋱⋮tp1⋯tpmT


## 4.2. Basic Flow

The main steps of the process of the proposed recommendation model are shown as follows: (1) compute the word vectors based on Word2vec; (2) computing the similarity of word vectors; (3) generating the tag similar matrix; (4) obtaining the worker-tag matrix and task-tag matrix; (5) computing the worker-task similarity matrix; (6)L2 standardization and normalization; (7) tasks and workers recommendation. Tag similar matrix generation uses Word2vec tool. Worker-task similarity computation uses mathematical methods introduced in the previous section. The section mainly introduces standardization and normalization method.L 2 standardization method: the L2 norm definition of vector x(x1,x2,…,xn) is shown as follows: norm(x)=x12+x22+⋯+xn2.In order to makex normalized to the unit L2 norm, the mapping between x and x′ is established, so that the L2 norm of x′ is 1, and the proof is shown as follows:(8)1=normx′=x12+x22+⋯+xn2normx=x1′2+x2′2+⋯+xn′2=x1normx2+x2normx2+⋯+xnnormx2where the value of xi′ is shown by (9)xi′=xinormxIn order to get the standardization and generality of data, the standardization data of L2 is normalized, so that the data fall in the interval [0,1], the conversion formula is shown by (10), where min(X) means the minimum in X, and max(X) is the maximum in X.(10)xi′=xi-min⁡Xmax⁡X-min⁡X


## 5. Experiment and Simulation

In this section, we conduct the comparison experiments on the simulation dataset and real dataset, respectively. The real dataset is the dataset crawled from Tianpeng web site.In the experiment, text8 is corpora training set, and experimental environment is Intel Core (TM) i5-337U CPU @1.8GHz dual-core, and 8GB memory.


### 5.1. The Experiments Conducted on Simulation Dataset

In this group of comparison experiments, the training parameters are shown in Table1.Table 1
Word2vec parameter setting.
Parameter Value Parameter Value window 8 hs 1 size 100 cbow yes threads 20 alpha 0.001 binary 0 negative 25In addition, the tag similar matrix after training is shown in Table2. In the matrix, the elements indicate the similarities between tags.Table 2
Tag similar matrix L of simulation dataset.
L1 L2 L3 L4 L5 L6 L7 … L1 1.000 0.407 0.124 0.119 0.126 0.434 0.075 … L2 0.407 1.000 0.766 0.917 0.993 0.642 0.546 … L3 0.124 0.766 1.000 0.930 0.477 0.526 0.744 … L4 0.119 0.917 0.930 1.000 0.909 0.531 0.394 … L5 0.126 0.993 0.477 0.909 1.000 0.636 0.860 … L6 0.434 0.642 0.526 0.531 0.636 1.000 0.166 … L7 0.075 0.546 0.744 0.394 0.860 0.166 1.000 … … … … … … … … … …In this group of experiments, there are 100 workers, 50 tasks, 2000 tags in the experiment. The worker-tag matrix is generated randomly, which is shown in Table3. The elements in Table 3 represent the similarities between workers and tags. The task-tag matrix is shown in Table 4. The elements in Table 4 indicate the similarities between tasks and tags. After computing the worker-task matrix, the standardization and normalization of worker-task matrix are shown in Table 5. The elements in Table 5 mean the similarities between workers and tasks.Table 3
Worker-tag matrix W.
L1 L2 L3 L4 L5 L6 L7 L8 L9 L10 L11 … W1 0 0 0 1 0 0 0 0 0 0 0 … W2 0 0 0 0 0 0 0 1 0 0 0 … W3 0 1 0 0 1 0 1 0 0 1 0 … W4 1 0 0 0 0 0 0 0 1 0 0 … W5 0 0 0 0 0 1 0 0 0 0 0 … … … … … … … … … … … … … …Table 4
Task-tag matrix T.
L1 L2 L3 L4 L5 L6 L7 L8 L9 L10 L11 … T1 0 0 0 0 0 0 0 1 0 0 0 … T2 0 0 1 0 0 0 1 0 1 0 0 … T3 0 0 0 0 0 0 0 0 0 0 0 … T4 0 0 1 0 1 0 0 0 0 0 0 … T5 0 0 0 0 0 0 1 0 0 0 0 … … … … … … … … … … … … … …Table 5
Worker-task similar matrix.
T1 T2 T3 T4 T5 T6 T7 … U1 0.754 0.410 0.369 0.438 0.365 0.420 0.396 … U2 0.680 0.694 0.712 0.706 0.682 0.747 0.720 … U3 0.387 0.378 0.407 0.403 0.385 0.678 0.351 … U4 0.405 0.304 0.681 0.731 0.733 0.835 0.704 … U5 0.279 0.278 0.696 0.284 0.294 0.265 0.324 … … … … … … … … … …Recall, precision, and F-measure are commonly used evaluation indexes [29]. The computing methods for the three evaluation indexes are shown by (11), (12), and (13). According to (11), (12), and (13), it can be seen that F-measure index is the comprehensive measure index through considering both recall and precision.(11)Recall=thequantityofrelatedinformationretrievedthequantityofrelatedinformationinsystem(12)Precision=thequantityofrelatedinformationretrievedthequantityofallinformationretrieved(13)F-measure=2×Precision×PecallPrecision+RecallThe threshold values are 0.55, 0.6, and 0.65, respectively, and the recall, precision, and F-measure of the 50 tasks are obtained. The comparison experimental results on recall, precision, and F-measure indexes are shown by Figures 3, 4, and 5, respectively. In these experiments, x-coordinate indicates the Task-tag matrix T, and y-coordinates are recall rate, precision rate, and F-measure rate, respectively. From the experimental results, it can be seen thatthreshold=0.6 has better performance than other two thresholds comprehensively.Figure 3
Recall of different thresholds.Figure 4
Precision of different thresholds.Figure 5
F-measure of different thresholds.In addition, we compare the proposed method with the method of tasks research. The experimental result is shown in Figure6, where x-coordinate indicates the Task-tag matrix T and y-coordinate means the number of workers. The method used in this paper is better than the method used in tasks research, which proves the effectiveness of the method of this paper. In addition, the potential workers can be found by lowering the threshold, which can be used to analyze the potential users.Figure 6
Comparison of experimental results.


### 5.2. The Experiments Conducted on Tianpeng Dataset

The data collected from the Tianpeng web site were collected to form a corpus for training, and the tag similarity matrix was obtained as shown in the Table6.Table 6
Tag similar matrix L of Tianpeng dataset.
L1 L2 L3 L4 L5 L6 L7 L8 L9 L10 L11 L12 L13 … L1 1.000 0.004 -0.041 0.100 0.018 0.048 0.020 -0.040 -0.049 -0.029 -0.009 0.038 -0.026 … L2 0.004 1.000 0.803 0.261 0.882 0.225 0.493 0.610 0.391 0.315 0.817 0.666 0.601 … L3 -0.041 0.803 1.000 0.231 0.761 0.166 0.393 0.533 0.351 0.259 0.722 0.603 0.609 … L4 0.100 0.261 0.231 1.000 0.268 0.134 0.234 0.176 0.191 0.229 0.248 0.251 0.173 … L5 0.018 0.882 0.761 0.268 1.000 0.218 0.475 0.571 0.352 0.228 0.753 0.659 0.583 … L6 0.048 0.225 0.166 0.134 0.218 1.000 0.133 0.135 0.101 0.095 0.198 0.222 0.181 … L7 0.020 0.493 0.393 0.234 0.475 0.133 1.000 0.334 0.190 0.192 0.504 0.459 0.296 … L8 -0.040 0.610 0.533 0.176 0.571 0.135 0.334 1.000 0.295 0.258 0.556 0.480 0.515 … L9 -0.049 0.391 0.351 0.191 0.352 0.101 0.190 0.295 1.000 0.248 0.386 0.239 0.277 … L10 -0.029 0.315 0.259 0.229 0.228 0.095 0.192 0.258 0.248 1.000 0.238 0.288 0.236 … L11 -0.009 0.817 0.722 0.248 0.753 0.198 0.504 0.556 0.386 0.238 1.000 0.616 0.535 … L12 0.038 0.666 0.603 0.251 0.659 0.222 0.459 0.480 0.239 0.288 0.616 1.000 0.455 … L13 -0.026 0.601 0.609 0.173 0.583 0.181 0.296 0.515 0.277 0.236 0.535 0.455 1.000 … … … … … … … … … … … … … … … …We select 510 workers and 371 tasks from Tianpeng dataset as experimental objects. Utilizing the dataset, we conduct the comparison experiments to verify the effectiveness of the proposed model. In the comparison experiments, 0.6 is taken as the threshold, and 20 tasks are randomly selected as recommended objects. The experimental results were compared with binary map matching and greedy algorithm in terms of recall rate, accuracy rate, and F-value measure indexes.According to the recall measure index, the comparison experimental result is shown by Figure7. The x-coordinate indicates the Task-tag matrix T, and y-coordinate presents the recall rate. From the experimental result, it can be seen that the proposed recommendation model has the best performance on recall rate through compared with greedy algorithm and bipartite graph matching. In addition, the proposed recommendation model has better stability with the changing of T.Figure 7
Recall of different methods.Figure8 shows the experimental result on precision rate. Similarly, the x-coordinate indicates the Task-tag matrix T, and y-coordinate means the precision rate. In experimental result, the average precision rate of the proposed recommendation is better than other two algorithms. From Figure 7, it can be seen that the proposed recommendation has the best performance on precision rate through compared with greedy algorithm and bipartite graph matching.Figure 8
Precision of different methods.According to the experimental result on F-measure shown by Figure9, we can see that the proposed recommendation also has the best performance on F-measure. In addition, F-measure index is the comprehensive measure index through considering both recall and precision. Therefore, we can infer that the proposed recommendation has the best performance through compared with greedy algorithm and bipartite graph matching algorithm.Figure 9
F-measure of different methods.Through the comparison shows that the proposed methods than the binary map matching method, greedy algorithm in the recall, F-measure index significantly, in terms of accuracy with high and low, because to make the task would be able to complete the task of recommended for workers as much as possible, including the potential of workers, so the accuracy index can be put lower in the recommended requirements. It can be seen that the method proposed in this paper has higher practical significance and application value.


## 5.1. The Experiments Conducted on Simulation Dataset

In this group of comparison experiments, the training parameters are shown in Table1.Table 1
Word2vec parameter setting.
Parameter Value Parameter Value window 8 hs 1 size 100 cbow yes threads 20 alpha 0.001 binary 0 negative 25In addition, the tag similar matrix after training is shown in Table2. In the matrix, the elements indicate the similarities between tags.Table 2
Tag similar matrix L of simulation dataset.
L1 L2 L3 L4 L5 L6 L7 … L1 1.000 0.407 0.124 0.119 0.126 0.434 0.075 … L2 0.407 1.000 0.766 0.917 0.993 0.642 0.546 … L3 0.124 0.766 1.000 0.930 0.477 0.526 0.744 … L4 0.119 0.917 0.930 1.000 0.909 0.531 0.394 … L5 0.126 0.993 0.477 0.909 1.000 0.636 0.860 … L6 0.434 0.642 0.526 0.531 0.636 1.000 0.166 … L7 0.075 0.546 0.744 0.394 0.860 0.166 1.000 … … … … … … … … … …In this group of experiments, there are 100 workers, 50 tasks, 2000 tags in the experiment. The worker-tag matrix is generated randomly, which is shown in Table3. The elements in Table 3 represent the similarities between workers and tags. The task-tag matrix is shown in Table 4. The elements in Table 4 indicate the similarities between tasks and tags. After computing the worker-task matrix, the standardization and normalization of worker-task matrix are shown in Table 5. The elements in Table 5 mean the similarities between workers and tasks.Table 3
Worker-tag matrix W.
L1 L2 L3 L4 L5 L6 L7 L8 L9 L10 L11 … W1 0 0 0 1 0 0 0 0 0 0 0 … W2 0 0 0 0 0 0 0 1 0 0 0 … W3 0 1 0 0 1 0 1 0 0 1 0 … W4 1 0 0 0 0 0 0 0 1 0 0 … W5 0 0 0 0 0 1 0 0 0 0 0 … … … … … … … … … … … … … …Table 4
Task-tag matrix T.
L1 L2 L3 L4 L5 L6 L7 L8 L9 L10 L11 … T1 0 0 0 0 0 0 0 1 0 0 0 … T2 0 0 1 0 0 0 1 0 1 0 0 … T3 0 0 0 0 0 0 0 0 0 0 0 … T4 0 0 1 0 1 0 0 0 0 0 0 … T5 0 0 0 0 0 0 1 0 0 0 0 … … … … … … … … … … … … … …Table 5
Worker-task similar matrix.
T1 T2 T3 T4 T5 T6 T7 … U1 0.754 0.410 0.369 0.438 0.365 0.420 0.396 … U2 0.680 0.694 0.712 0.706 0.682 0.747 0.720 … U3 0.387 0.378 0.407 0.403 0.385 0.678 0.351 … U4 0.405 0.304 0.681 0.731 0.733 0.835 0.704 … U5 0.279 0.278 0.696 0.284 0.294 0.265 0.324 … … … … … … … … … …Recall, precision, and F-measure are commonly used evaluation indexes [29]. The computing methods for the three evaluation indexes are shown by (11), (12), and (13). According to (11), (12), and (13), it can be seen that F-measure index is the comprehensive measure index through considering both recall and precision.(11)Recall=thequantityofrelatedinformationretrievedthequantityofrelatedinformationinsystem(12)Precision=thequantityofrelatedinformationretrievedthequantityofallinformationretrieved(13)F-measure=2×Precision×PecallPrecision+RecallThe threshold values are 0.55, 0.6, and 0.65, respectively, and the recall, precision, and F-measure of the 50 tasks are obtained. The comparison experimental results on recall, precision, and F-measure indexes are shown by Figures 3, 4, and 5, respectively. In these experiments, x-coordinate indicates the Task-tag matrix T, and y-coordinates are recall rate, precision rate, and F-measure rate, respectively. From the experimental results, it can be seen thatthreshold=0.6 has better performance than other two thresholds comprehensively.Figure 3
Recall of different thresholds.Figure 4
Precision of different thresholds.Figure 5
F-measure of different thresholds.In addition, we compare the proposed method with the method of tasks research. The experimental result is shown in Figure6, where x-coordinate indicates the Task-tag matrix T and y-coordinate means the number of workers. The method used in this paper is better than the method used in tasks research, which proves the effectiveness of the method of this paper. In addition, the potential workers can be found by lowering the threshold, which can be used to analyze the potential users.Figure 6
Comparison of experimental results.


## 5.2. The Experiments Conducted on Tianpeng Dataset

The data collected from the Tianpeng web site were collected to form a corpus for training, and the tag similarity matrix was obtained as shown in the Table6.Table 6
Tag similar matrix L of Tianpeng dataset.
L1 L2 L3 L4 L5 L6 L7 L8 L9 L10 L11 L12 L13 … L1 1.000 0.004 -0.041 0.100 0.018 0.048 0.020 -0.040 -0.049 -0.029 -0.009 0.038 -0.026 … L2 0.004 1.000 0.803 0.261 0.882 0.225 0.493 0.610 0.391 0.315 0.817 0.666 0.601 … L3 -0.041 0.803 1.000 0.231 0.761 0.166 0.393 0.533 0.351 0.259 0.722 0.603 0.609 … L4 0.100 0.261 0.231 1.000 0.268 0.134 0.234 0.176 0.191 0.229 0.248 0.251 0.173 … L5 0.018 0.882 0.761 0.268 1.000 0.218 0.475 0.571 0.352 0.228 0.753 0.659 0.583 … L6 0.048 0.225 0.166 0.134 0.218 1.000 0.133 0.135 0.101 0.095 0.198 0.222 0.181 … L7 0.020 0.493 0.393 0.234 0.475 0.133 1.000 0.334 0.190 0.192 0.504 0.459 0.296 … L8 -0.040 0.610 0.533 0.176 0.571 0.135 0.334 1.000 0.295 0.258 0.556 0.480 0.515 … L9 -0.049 0.391 0.351 0.191 0.352 0.101 0.190 0.295 1.000 0.248 0.386 0.239 0.277 … L10 -0.029 0.315 0.259 0.229 0.228 0.095 0.192 0.258 0.248 1.000 0.238 0.288 0.236 … L11 -0.009 0.817 0.722 0.248 0.753 0.198 0.504 0.556 0.386 0.238 1.000 0.616 0.535 … L12 0.038 0.666 0.603 0.251 0.659 0.222 0.459 0.480 0.239 0.288 0.616 1.000 0.455 … L13 -0.026 0.601 0.609 0.173 0.583 0.181 0.296 0.515 0.277 0.236 0.535 0.455 1.000 … … … … … … … … … … … … … … … …We select 510 workers and 371 tasks from Tianpeng dataset as experimental objects. Utilizing the dataset, we conduct the comparison experiments to verify the effectiveness of the proposed model. In the comparison experiments, 0.6 is taken as the threshold, and 20 tasks are randomly selected as recommended objects. The experimental results were compared with binary map matching and greedy algorithm in terms of recall rate, accuracy rate, and F-value measure indexes.According to the recall measure index, the comparison experimental result is shown by Figure7. The x-coordinate indicates the Task-tag matrix T, and y-coordinate presents the recall rate. From the experimental result, it can be seen that the proposed recommendation model has the best performance on recall rate through compared with greedy algorithm and bipartite graph matching. In addition, the proposed recommendation model has better stability with the changing of T.Figure 7
Recall of different methods.Figure8 shows the experimental result on precision rate. Similarly, the x-coordinate indicates the Task-tag matrix T, and y-coordinate means the precision rate. In experimental result, the average precision rate of the proposed recommendation is better than other two algorithms. From Figure 7, it can be seen that the proposed recommendation has the best performance on precision rate through compared with greedy algorithm and bipartite graph matching.Figure 8
Precision of different methods.According to the experimental result on F-measure shown by Figure9, we can see that the proposed recommendation also has the best performance on F-measure. In addition, F-measure index is the comprehensive measure index through considering both recall and precision. Therefore, we can infer that the proposed recommendation has the best performance through compared with greedy algorithm and bipartite graph matching algorithm.Figure 9
F-measure of different methods.Through the comparison shows that the proposed methods than the binary map matching method, greedy algorithm in the recall, F-measure index significantly, in terms of accuracy with high and low, because to make the task would be able to complete the task of recommended for workers as much as possible, including the potential of workers, so the accuracy index can be put lower in the recommended requirements. It can be seen that the method proposed in this paper has higher practical significance and application value.


## 6. Conclusion

Crowdsourcing is the prefect shown of group wisdom. It was applied in many fields as a new business model. In recent years, it has become the new hot research in computer science. The success key of crowdsourcing is to recommend task to appropriate worker. The recommendation method based on tag similar matrix is proposed in this paper. The method uses Word2vec technology to generate tag similar matrix and then computes the similarity of worker and task. According to the comparison experiments, it proves that the method is effective and feasible. The recommendation method can be extended to other fields with the different corpora.Because the success key of crowdsourcing is the participate rate of workers, it has become a hot topic in crowdsourcing research, such as reputation mechanism, preference evolution, and privacy protection of workers. It will be the focus of future research to improve the accuracy of recommender systems by combining recommender systems with reputation, preference evolution and historical information.



---

*Source: 2121850-2019-03-24.xml*