# A Cyber Physical System Crowdsourcing Inference Method Based on Tempering: An Advancement in Artificial Intelligence Algorithms

Authors: Jia Liu; Mingchu Li; William C. Tang; Sardar M. N. Islam
Journal: Wireless Communications and Mobile Computing
 (2021)

---

## Abstract

Activity selection is critical for the smart environment and Cyber-Physical Systems (CPSs) that can provide timely and intelligent services, especially as the number of connected devices is increasing at an unprecedented speed. As it is important to collect labels by various agents in the CPSs, crowdsourcing inference algorithms are designed to help acquire accurate labels that involve high-level knowledge. However, there are some limitations in the algorithm in the existing literature such as incurring extra budget for the existing algorithms, inability to scale appropriately, requiring the knowledge of prior distribution, difficulties to implement these algorithms, or generating local optima. In this paper, we provide a crowdsourcing inference method with variational tempering that obtains ground truth as well as considers both the reliability of workers and the difficulty level of the tasks and ensure a local optimum. The numerical experiments of the real-world data indicate that our novel variational tempering inference algorithm performs better than the existing advancing algorithms. Therefore, this paper provides a new efficient algorithm in CPSs and machine learning, and thus, it makes a new contribution to the literature.


---

## Body

## 1. Introduction

### 1.1. Research Background

Crowdsourcing refers to the practice whereby a company or organization outsources the tasks that used to be performed by employees to the crowds of nonspecific Internet agents. The wide use of the Internet enables crowdsourcing to make use of the wisdom of crowds as a cheap, fast, and convenient method. Therefore, crowdsourcing is much more powerful than traditional methods, especially in the fields of computer vision, natural language processing, environmental protection [1, 2], etc.In recent years, the development of embedded computing and wireless communications has enabled the CPSs to become an important research and industrial field. A CPS is often designed as a cooperative network, consisting of sensors, actuators, and controllers [3–6]. In many applications, such as object identification, traffic management, and smart health, CPSs need to extract information and process data in a large scale, a task that is difficult or unable to accomplish at one single device [7, 8]. Therefore, crowdsourcing is important for CPSs to combat the above limitations [9–11].However, existing CPS crowdsourcing also has disadvantages: for instance, (1) sometimes, agents can be unreliable, or there may be attackers or spammers who aim to corrupt the system; therefore, it is intuitive to assign the same tasks to multiple workers; (2) the difficulty levels of the tasks can vary a lot that if the workers are not properly rewarded, they may choose only the easy tasks, thus degrading the system; (3) it is also critical to allocate tasks to the right kind of workers in order to improve the efficiency and quality of the answers. However, when there is no longer a need to improve the quality of the collected data, we need to improve the computing algorithm to better extract the useful information from the data. Therefore, it is very important to implement inference algorithms to extract information from the data and deal with the above three disadvantages, the procedure of which is called crowdsourcing inference problem.


### 1.2. Related Work

In many fields, such as human pose estimation and smart medical, crowdsourcing inference has become a useful and cost-effective method to denote a large quantity of data. The existing crowdsourcing inference algorithms can be classified into four categories, including the weighted majority voting (WMV) method, the statistical method, new kind of inference tools, and variational method.The first way to obtain ground truth is by majority voting (MV) or WMV, assigning different workers’ different weight according to their expertise or reliability [12–15]. The limitation of MV is that it regards all workers to be equally and easily subject to attacks. In contrast to MV, WMV is intuitively more efficient, and all other inference algorithms can be treated as a kind of WMV, so it is meaningless to discuss the limitations of WMV since the situation varies accordingly. In general, the weight of workers should be assigned according to their reliability; therefore, some researchers use qualification test or hidden test to determine the weight of workers [16]. However, this approach is highly dependent on a rich budget and the cooperation of workers.Another way is to use a statistical method, for example, maximum likelihood estimation (MLE) and expectation maximization (EM). Expectation maximization (EM) is a well-established way to compute the hyperparameters and ground truth in the process of implementing MLE. EM first makes some initial guess on the hyperparameters, then computes expectations under these hyperparameters, and repeats this process until convergence is reached. This process can be treated as an iterative decoding process, so EM is convenient and easy to understand [16–18]. However, EM may result in local optima and is challenging to scale.Therefore, some researchers turn to a new kind of inference tools, including the widely used back-propagation (BP) and mean filed method (MF) [19]. These algorithms can be effective, as well as can guarantee local optimality and sometimes can guarantee global optimality. However, standard BP requires the knowledge of prior distribution, and in reality, it is difficult to implement due to the process of passing messages in the form of sufficient statistics. MF is the approximation approach that maximizes energy functional over the approximate distribution, which is easy to compute. However, MF may lose valuable information and fail to capture the dependency property of posterior distribution.An efficient way to avoid the above disadvantages is to use variational method. For example, Liu et al.’s [20] variational inference algorithm (VMP) performs extremely well with good worker distribution prior. In tasks of crowdsourcing inference, Hoffman et al. [21] have formulated it into a stochastic variational inference problem. Liu et al. [22] also formulated the crowdsourcing inference problem into a variational inference model. Cai et al. [23] proposed a crowdsourcing prediction algorithm using variation inference. In summary, variational inference (VI) translates inference problem into an optimization problem with multiple local optimum. However, using the tempering technique, we can curb this to some extent. Intuitively, VI objective uses high-entropy variational distribution to replace data-proper variational distribution. By tempering, we can punish low-entropy distributions and loose this restraint gradually to reach distributions that fit the data.


### 1.3. Motivation

In view of the problems of the existing literature in this area of CPS crowdsourcing inference algorithms discussed above, the main objectives of this paper are to model the CPS crowdsourcing inference problem into a message passing algorithm, i.e., our novel variational tempering inference algorithm (VTI), and to study the performance error bound of this algorithm.As a popular sampling as well as computing method, Markov Chain Monte Carlo (MCMC) has proved its efficiency in the field of CPSs and big data, due to its ability to deal with large amount of data, and simplifies the gradient computation [24]. Variational tempering (VT) is an extension of MCMC. Therefore, in VTI, we introduce global temperature. We then implement the traditional VI method to compute the gradient of it in each iteration and further update it in the next iteration, until the algorithm reaches convergence, which is also similar to the way that we treat tasks’ answers as a distribution as well as worker reliability.After researching into the existing works, we borrow the idea of VT and solve the problem of crowdsourcing inference by considering worker reliability and task difficulty. There are few papers in the literature considering task difficulty level. The reason we are considering task difficulty level in this paper is that we can assign tasks in a more efficient way [25], infer the ground truth more accurately [26, 27], and reward workers by means of incentives [28] etc. Therefore, the main objective of this paper is to obtain task ground truth by making the best use of worker reliability and task difficulty. The specific tasks we will perform in this paper are the following: (1) Model the crowdsourcing inference problem into a message passing algorithm. (2) Study the performance error bound of our novel variational tempering inference algorithm (VTI). (3) Use real data to simulate the probability error of VTI.


### 1.4. Contributions

The main contribution of this paper is a new algorithm to CPS crowdsourcing inference. Based on variational analysis and the property of exponential family, a kind of probabilistic graphical model, we try to solve crowdsourcing inference problem in CPSs. First, we infer the ground truth by considering worker reliability. Being different from previous works, we then hypothesize that tasks have different difficulty levels and use this as a parameter to eliminate the same worker constraint.Therefore, the contributions of the paper are listed as follows:(1)
Formulate the CPS crowdsourcing inference problem into a message passing algorithm(2)
Study the performance error bound of the proposed variational tempering inference algorithm (VTI)(3)
Use real data to simulate the probability error of VTI. In general cases, VTI performs better than other algorithmsIn this paper, we first review CPS crowdsourcing inference and variational tempering in Section1. We then define the background assumptions of the inference model in Section 2. Thereafter, we derive our algorithm stepwise and show the procedure of how we obtain the results in Sections 3 and 4. We show the results of our algorithm in comparison with others in Section 5. Our VTI algorithm requires less constraint on worker prior and task difficulty level and performs better than the existing inference algorithms. In the end, Section 6 presents the conclusion of this study.


## 1.1. Research Background

Crowdsourcing refers to the practice whereby a company or organization outsources the tasks that used to be performed by employees to the crowds of nonspecific Internet agents. The wide use of the Internet enables crowdsourcing to make use of the wisdom of crowds as a cheap, fast, and convenient method. Therefore, crowdsourcing is much more powerful than traditional methods, especially in the fields of computer vision, natural language processing, environmental protection [1, 2], etc.In recent years, the development of embedded computing and wireless communications has enabled the CPSs to become an important research and industrial field. A CPS is often designed as a cooperative network, consisting of sensors, actuators, and controllers [3–6]. In many applications, such as object identification, traffic management, and smart health, CPSs need to extract information and process data in a large scale, a task that is difficult or unable to accomplish at one single device [7, 8]. Therefore, crowdsourcing is important for CPSs to combat the above limitations [9–11].However, existing CPS crowdsourcing also has disadvantages: for instance, (1) sometimes, agents can be unreliable, or there may be attackers or spammers who aim to corrupt the system; therefore, it is intuitive to assign the same tasks to multiple workers; (2) the difficulty levels of the tasks can vary a lot that if the workers are not properly rewarded, they may choose only the easy tasks, thus degrading the system; (3) it is also critical to allocate tasks to the right kind of workers in order to improve the efficiency and quality of the answers. However, when there is no longer a need to improve the quality of the collected data, we need to improve the computing algorithm to better extract the useful information from the data. Therefore, it is very important to implement inference algorithms to extract information from the data and deal with the above three disadvantages, the procedure of which is called crowdsourcing inference problem.


## 1.2. Related Work

In many fields, such as human pose estimation and smart medical, crowdsourcing inference has become a useful and cost-effective method to denote a large quantity of data. The existing crowdsourcing inference algorithms can be classified into four categories, including the weighted majority voting (WMV) method, the statistical method, new kind of inference tools, and variational method.The first way to obtain ground truth is by majority voting (MV) or WMV, assigning different workers’ different weight according to their expertise or reliability [12–15]. The limitation of MV is that it regards all workers to be equally and easily subject to attacks. In contrast to MV, WMV is intuitively more efficient, and all other inference algorithms can be treated as a kind of WMV, so it is meaningless to discuss the limitations of WMV since the situation varies accordingly. In general, the weight of workers should be assigned according to their reliability; therefore, some researchers use qualification test or hidden test to determine the weight of workers [16]. However, this approach is highly dependent on a rich budget and the cooperation of workers.Another way is to use a statistical method, for example, maximum likelihood estimation (MLE) and expectation maximization (EM). Expectation maximization (EM) is a well-established way to compute the hyperparameters and ground truth in the process of implementing MLE. EM first makes some initial guess on the hyperparameters, then computes expectations under these hyperparameters, and repeats this process until convergence is reached. This process can be treated as an iterative decoding process, so EM is convenient and easy to understand [16–18]. However, EM may result in local optima and is challenging to scale.Therefore, some researchers turn to a new kind of inference tools, including the widely used back-propagation (BP) and mean filed method (MF) [19]. These algorithms can be effective, as well as can guarantee local optimality and sometimes can guarantee global optimality. However, standard BP requires the knowledge of prior distribution, and in reality, it is difficult to implement due to the process of passing messages in the form of sufficient statistics. MF is the approximation approach that maximizes energy functional over the approximate distribution, which is easy to compute. However, MF may lose valuable information and fail to capture the dependency property of posterior distribution.An efficient way to avoid the above disadvantages is to use variational method. For example, Liu et al.’s [20] variational inference algorithm (VMP) performs extremely well with good worker distribution prior. In tasks of crowdsourcing inference, Hoffman et al. [21] have formulated it into a stochastic variational inference problem. Liu et al. [22] also formulated the crowdsourcing inference problem into a variational inference model. Cai et al. [23] proposed a crowdsourcing prediction algorithm using variation inference. In summary, variational inference (VI) translates inference problem into an optimization problem with multiple local optimum. However, using the tempering technique, we can curb this to some extent. Intuitively, VI objective uses high-entropy variational distribution to replace data-proper variational distribution. By tempering, we can punish low-entropy distributions and loose this restraint gradually to reach distributions that fit the data.


## 1.3. Motivation

In view of the problems of the existing literature in this area of CPS crowdsourcing inference algorithms discussed above, the main objectives of this paper are to model the CPS crowdsourcing inference problem into a message passing algorithm, i.e., our novel variational tempering inference algorithm (VTI), and to study the performance error bound of this algorithm.As a popular sampling as well as computing method, Markov Chain Monte Carlo (MCMC) has proved its efficiency in the field of CPSs and big data, due to its ability to deal with large amount of data, and simplifies the gradient computation [24]. Variational tempering (VT) is an extension of MCMC. Therefore, in VTI, we introduce global temperature. We then implement the traditional VI method to compute the gradient of it in each iteration and further update it in the next iteration, until the algorithm reaches convergence, which is also similar to the way that we treat tasks’ answers as a distribution as well as worker reliability.After researching into the existing works, we borrow the idea of VT and solve the problem of crowdsourcing inference by considering worker reliability and task difficulty. There are few papers in the literature considering task difficulty level. The reason we are considering task difficulty level in this paper is that we can assign tasks in a more efficient way [25], infer the ground truth more accurately [26, 27], and reward workers by means of incentives [28] etc. Therefore, the main objective of this paper is to obtain task ground truth by making the best use of worker reliability and task difficulty. The specific tasks we will perform in this paper are the following: (1) Model the crowdsourcing inference problem into a message passing algorithm. (2) Study the performance error bound of our novel variational tempering inference algorithm (VTI). (3) Use real data to simulate the probability error of VTI.


## 1.4. Contributions

The main contribution of this paper is a new algorithm to CPS crowdsourcing inference. Based on variational analysis and the property of exponential family, a kind of probabilistic graphical model, we try to solve crowdsourcing inference problem in CPSs. First, we infer the ground truth by considering worker reliability. Being different from previous works, we then hypothesize that tasks have different difficulty levels and use this as a parameter to eliminate the same worker constraint.Therefore, the contributions of the paper are listed as follows:(1)
Formulate the CPS crowdsourcing inference problem into a message passing algorithm(2)
Study the performance error bound of the proposed variational tempering inference algorithm (VTI)(3)
Use real data to simulate the probability error of VTI. In general cases, VTI performs better than other algorithmsIn this paper, we first review CPS crowdsourcing inference and variational tempering in Section1. We then define the background assumptions of the inference model in Section 2. Thereafter, we derive our algorithm stepwise and show the procedure of how we obtain the results in Sections 3 and 4. We show the results of our algorithm in comparison with others in Section 5. Our VTI algorithm requires less constraint on worker prior and task difficulty level and performs better than the existing inference algorithms. In the end, Section 6 presents the conclusion of this study.


## 2. Preliminaries

In this part, we describe the mathematical notations of VTI. Assuming in the crowdsourcing network, there areM workers and N tasks. Suppose that each task has the value of ±1, and we use zi to denote the true label of it, i.e., zi∈±1. We then denote Nj as worker j’s neighbor set, i.e., the indexes of tasks that worker j takes. Likewise, Mi task i’s neighbor set, i.e., the indexes of workers that task i is assigned to. L∈0,±1N×M is the label matrix, and Lij∈±1 is worker j’s answer or label to task i, and Lij=0 means there is no link between worker j and task i. The goal of our VTI is to get the ground truth and difficulty level of task i as well as the reliability of worker j.We useqj=probLij=zi,i∈Ni to denote the reliability of worker j and di=probLij=zi,j∈Mj to denote the difficulty level of task i. Note that q and d are two separate factors, so the calculation of q does not involve d, and the calculation of d does not involve q. However, they both influence the label of each task. In addition, when qj≈1, worker j produces 100% right answers. qj≈1/2 can be tricky since the worker may be a spammer or just submits random answers that contain no useful information. However, if qj≤1/2, the worker’s answers can be made use of by just reverting its answer matrix. Further, we assume each worker is independent, and its reliability is drawn from the same distribution with hyperparameter θ1. We assume different tasks are also independent, drawn from the distribution with hyperparameter θ2.It is reasonable to study cases when there are fewer spammers and attackers in the system; therefore, we assumeEqj∣θ1≥1/2. We use beta prior for worker reliability pqj∣θ1∝qjα1−11−qjβ1−1 [29].In Liu et al.’s work [22], the authors treat the crowdsourcing issue as a graphical inference problem, of which they first use Bayesian theory to reduce the objective function and then use variational inference to compute
(1)pz,q,d∣L,θ∝∏j∈Mpqj∣θ1∏i∈Npdi∣θ2∏i∈NjpLij∣zi,qj,di=∏j∈Mpqj∣θ1qjcj1−qjγj−cj∏i∈Npdi∣θ2,whereγj=Nj is the number of answers given by worker j and cj≔∑i∈NjILij=zi is the number of correct answers worker j gives. It follows that the value of z with the minimum error probability is
(2)z^i=argmaxzipzi∣L,θwherepzi∣L,θ=∑zN\i∫d∫qpz,q,d∣L,θdqddNote that variablez is discrete and q is continuous, and θ=θ1,θ2. Because the joint probability depends on both q and z, it is intractable to calculate pz,q∣L,θ directly. We can take a detour and take the integration over qj, resulting in a marginal posterior over z.
(3)pz∣L,θ=∫d∫qpz,q,d∣L,θdqdd=∏j∈M∫01∫01pqj∣θ1pdi∣θ2qjcj1−qjγj−cjdqdd=def∏j∈MψjzNj∏i∈NφizNiof which ψjzNj is the local factor of worker j, i.e., the summation of all tasks taken by worker j; φizNi is the local factor of task i, i.e., the summation of all workers take task i. Here, we also assume that ψj on θ1 and L are independent, and φi on θ2 and L are independent.Further, we use Markov random field assumption to modelpz∣L,θ and treat the task assignment graph as a factor graph, of which variable nodes are task nodes, and factor nodes are worker nodes [30, 31]. We use mi→j to denote the messages passing from tasks to workers and mj→i to denote the messages passing from workers to tasks. Messages are initialized as independent Gaussian distribution or other suitable distribution. We then update them iteratively until convergence. To put it in a simple way, the messages are equivalent to the ground truth combined with all relevant factors. Therefore, bizi is the belief of the tasks, i.e., the intermediate estimated ground truth of the tasks. Therefore, the updating equations can be formulated as follows:
(4)Fromtaskstoworkers:mi→jt+1zi∝∑zNi\jφizNi∏i′∈Ni\jmj′→itzi,Fromworkerstotasks:mj→it+1zi∝∑zNj\iψjzNj∏i′∈Nj\imi′→jt+1zi′,Calculatingthebeliefs:bit+1zi∝∏j∈Mimj→it+1zi.Note that tasks can have binary or multiple dimension labels, and in this case,mi→j or mj→i can instead be vectors.


## 3. System Model

In this section, we introduce our VTI and then form its derivations. We use label aggregation problem as an example to study crowdsourcing problem, and since others can be transformed to a label model easily, we use label aggregation to show VTI algorithm. Intuitively, we use log error rate to indicate the performance of our algorithm and the comparison criteria between existing inference algorithms. Further, the convergence rate and speed are also important criteria for inference problems.In VTI, we add the task difficultyd to improve inference efficiency. At first, we can use d to help compute worker reliability q and then use both d and q to predict ground truth z. In general, there are three advantages when task requestors take into account of the task difficulty level. First, worker reliability parameter will be more valid if task difficulty is included to compute it. The reason is simple: more difficult tasks require workers with more expertise [32, 33]. Second, task difficulty is useful when assigning tasks and can reduce the task time and budget to some extent. Third, it is useful to consider task difficulty when compensating workers, which in turn can sustain the crowdsourcing system. Surely, it is important to consider cases where workers have no incentive to complete a task as soon as possible and may always submit completed tasks in the last minute. Some workers might even submit a random label in a short time. Therefore, a good difficulty level indicator should be able to deal with those situations.First, we denote workerj’s reliability as q=qj:j∈M, task difficulty as d=di:i∈N, and the true answer of tasks z=zi:i∈N. Therefore, the joint probability of worker reliability, task difficulty, ground truth, and conditioned on the answer matrix L and θ is as Equation (1).


### 3.1. Assumptions and ELBO Formation

In VTI, we use a three-layered graphical model to show the causal relationship of the parameters. In this model, all data points share global variables, while each local hidden variable belongs to each data point. As defined above, we denoteL=L1:N as observed variables (label vector of the tasks), z=z1:N as local hidden values, and θ as global hidden values. The joint density of the model is
(5)pq,d,z,L=pq,d∣θ∏i=1Npzi,Li∣d,qFigure1 illustrates the graphical model of VTI.Figure 1
Probability graphical model of VTI.Computing the posterior inference is the primary task of variational inference. Therefore, our main task is to estimate the ground truth of every task according to collected answers, which ispθ,z∣L, the conditional probability of hidden variables (ground truth), and global hidden variables (data distribution), given observations (collected answers from the crowdsourcing system). However, it is often the case that this conditional probability is difficult or computing-intensive to solve. So it is intuitive for us to resort to approximation methods.VI is the method of using a family of distributions that is easy to compute to approximate the desired objective posterior distribution. The direct way is to minimize the KL divergence between the posterior distribution and the approximate distribution, which is also a function of the hyperparameters. Thus, the computing process is to solve the optimum of a functional, which is called variational analysis. In the domain of machine learning (ML), this is called the procedure of solving evidence lower bound (ELBO) in terms of a variational parameter.(6)Lv=Eqlogpq,d,z,L−Eqlogqq,d,z∣vof which q=qq,d,z∣v is the variational distribution we choose to approximate the true joint distribution pq,d,z∣L. The detailed derivation of Equation (6) can be found in Appendix.Assuming that the Bayesian network of our crowdsourcing inference model is fully factored, i.e., each worker and task are independent, and using the chain rule to extend the joint probability, we have(7)qq,d,z∣v=qq∣λ1∏qzi∣ϕiqd∣λ2of which v is the general notation of the variational parameter, λ1 is the global variational variable for q, λ2 is the global variational variable for d, λ=λ1,λ2, and ϕ is local variational variable. VI optimizes Equation (2) using a gradient or coordinate ascent. To better find local optimum, we use variational inference with tempering.As mentioned before, the procedure of tempering can promote the performance of the optimization process by curbing multioptimum. Therefore, we apply it in our VTI. UsingT≥1 to denote the global temperature, the joint probability becomes
(8)pq,d,z,L∣T=pz,L∣q,d1/Tpq∣θ1pd∣θ2cTof which
(9)cT=∫pz,L∣q,d1/Tpq∣θ1pd∣θ2dLdzdqdd.Applying Equation (8) in (6), we reach the tempered ELBO.
(10)LAλ,ϕ;T=Eqlogpq∣θ1+Eqlogpd∣θ2−Eqlogqq,d∣λ+∑i=1NEqlogpLi,zi∣q,dT−Eqlogpzi∣ϕi.Note that the standard tempering method starts from high temperature and gradually reduces it, taking little account of the influence of data at first, reaching a high-entropy solution, and then slowly makes use of the data or evidence, and reaching a distribution more resembling the data.Now, we discuss why tempering is effective. The 1st and 3rd terms of Equation (10) are the expected log prior and the log-likelihood. By maximizing these terms, it can result in hidden variables that can better explain the data. The 2nd and 4th terms combined are the variational distribution entropy. Since entropy is convex, it can act as a regularizer which distributes into a hidden variable’s configuration. With smaller 1/T, we obtain a small likelihood of ground truth and smooth entropy distribution. By decreasing T gradually, our variational distribution puts more and more weight on observed data. According to Mandt et al. [34], there are two types of tempering method, one method tempers both likelihood and prior, and the other method tempers only the likelihood. The latter has no problem of gradient stuck during iteration.This paper discusses gradient-based tempering. As similar to a regular optimization problem, if the tempering is too slow, the algorithm will converge too fast and never reach the global optimum. If we temper too fast, on the other hand, we may skip the global optimum. Since it is difficult to find the proper schedule, we put our emphasis on learning data sequence adaptively. We use variational tempering, a method learning temperature schedule. In variational tempering, we introduce temperature as an auxiliary variable. Therefore, if the temperature is1, it will degenerate to the original variational inference. We consider finite discrete temperature, for example, 1=T1≤T2≤⋯≤TM, which allows us to compute partition function beforehand.


### 3.2. Objective Function

It is easy to derive thatpL,z,q,d,y=pL,z,q,d∣ypy. Given py=∏m=1Mπmym, the tempered joint of task difficulty, worker reliability, ground truth and worker labels can be written as
(11)pL,z,q,d∣y=pq,dcTy∏i=1NpLi,zi∣q,d1/Tyof whichTy is a temperature related to y. Thus, the model becomes
(12)pL,z,q,d,y=pq,d∏m=1MπmcTm∏i=1NpLi,zi∣q,d1/Tmym.The objective of VTI is to maximize the above joint probability. By further using the chain rule of Bayesian theory, we obtain(13)pz,q,d,y∣ψ,λ,γ=qz∣ψqq,d∣λqy∣γof which r denotes the variational parameter for the temperature. Therefore, we obtain the tempered ELBO of VTI.
(14)LTλ,ϕ;T=Eqlogpq+Eqlogpd+Eqlogpy−Eqlogqq−Eqlogqd−Eq1Ty∑i=1NEqlogpLi,zi∣q,d−EqlogCTy−∑i=1NEqlogqzi−Eqlogqy.Therefore, we can maximize this tempered ELBO, instead of optimizing the objective function directly, to obtain ground truthz and hidden parameters q,d.


### 3.3. Local Temperature

In real practice, often the data would arrive one at a time, or sometimes it is difficult to compute a huge amount of data point at once, so we use the incremental method, which computes each data point as soon as it arrives. This method can simplify the computation and show the influence of each data point to the whole network. Therefore, we can treat the potential or log value of each data point as the local factor. And instead of computing the global temperature at once, we can calculate their local temperature in a similar way. The algorithm is also more scalable to future data. Just like heat flows through the network with thermal conductivity, we can treat the potential or temperature of each local data that passes through the data network. Usingti to denote the local temperature, the joint probability of task labels, ground truth, hyperparameter, and temperature becomes
(15)pL,z,q,d,t∝pq,d∏i=1NpLi,zi∣q,d1/tipti.We can see from the above equation that adding the temperature parameter reduces the weight of the global hidden value and data, i.e., the weight of the ground truth and the task labels. In this way, the resulted distribution of the tasks will have higher entropy. Note that we no longer have to calculate the partition function since there is a∝ in the equation. By adjusting the local temperature, we can obtain a more desired outcome. For instance, outliers can be best demonstrated by higher temperature, while lower temperature can give us the main situation of the data.


## 3.1. Assumptions and ELBO Formation

In VTI, we use a three-layered graphical model to show the causal relationship of the parameters. In this model, all data points share global variables, while each local hidden variable belongs to each data point. As defined above, we denoteL=L1:N as observed variables (label vector of the tasks), z=z1:N as local hidden values, and θ as global hidden values. The joint density of the model is
(5)pq,d,z,L=pq,d∣θ∏i=1Npzi,Li∣d,qFigure1 illustrates the graphical model of VTI.Figure 1
Probability graphical model of VTI.Computing the posterior inference is the primary task of variational inference. Therefore, our main task is to estimate the ground truth of every task according to collected answers, which ispθ,z∣L, the conditional probability of hidden variables (ground truth), and global hidden variables (data distribution), given observations (collected answers from the crowdsourcing system). However, it is often the case that this conditional probability is difficult or computing-intensive to solve. So it is intuitive for us to resort to approximation methods.VI is the method of using a family of distributions that is easy to compute to approximate the desired objective posterior distribution. The direct way is to minimize the KL divergence between the posterior distribution and the approximate distribution, which is also a function of the hyperparameters. Thus, the computing process is to solve the optimum of a functional, which is called variational analysis. In the domain of machine learning (ML), this is called the procedure of solving evidence lower bound (ELBO) in terms of a variational parameter.(6)Lv=Eqlogpq,d,z,L−Eqlogqq,d,z∣vof which q=qq,d,z∣v is the variational distribution we choose to approximate the true joint distribution pq,d,z∣L. The detailed derivation of Equation (6) can be found in Appendix.Assuming that the Bayesian network of our crowdsourcing inference model is fully factored, i.e., each worker and task are independent, and using the chain rule to extend the joint probability, we have(7)qq,d,z∣v=qq∣λ1∏qzi∣ϕiqd∣λ2of which v is the general notation of the variational parameter, λ1 is the global variational variable for q, λ2 is the global variational variable for d, λ=λ1,λ2, and ϕ is local variational variable. VI optimizes Equation (2) using a gradient or coordinate ascent. To better find local optimum, we use variational inference with tempering.As mentioned before, the procedure of tempering can promote the performance of the optimization process by curbing multioptimum. Therefore, we apply it in our VTI. UsingT≥1 to denote the global temperature, the joint probability becomes
(8)pq,d,z,L∣T=pz,L∣q,d1/Tpq∣θ1pd∣θ2cTof which
(9)cT=∫pz,L∣q,d1/Tpq∣θ1pd∣θ2dLdzdqdd.Applying Equation (8) in (6), we reach the tempered ELBO.
(10)LAλ,ϕ;T=Eqlogpq∣θ1+Eqlogpd∣θ2−Eqlogqq,d∣λ+∑i=1NEqlogpLi,zi∣q,dT−Eqlogpzi∣ϕi.Note that the standard tempering method starts from high temperature and gradually reduces it, taking little account of the influence of data at first, reaching a high-entropy solution, and then slowly makes use of the data or evidence, and reaching a distribution more resembling the data.Now, we discuss why tempering is effective. The 1st and 3rd terms of Equation (10) are the expected log prior and the log-likelihood. By maximizing these terms, it can result in hidden variables that can better explain the data. The 2nd and 4th terms combined are the variational distribution entropy. Since entropy is convex, it can act as a regularizer which distributes into a hidden variable’s configuration. With smaller 1/T, we obtain a small likelihood of ground truth and smooth entropy distribution. By decreasing T gradually, our variational distribution puts more and more weight on observed data. According to Mandt et al. [34], there are two types of tempering method, one method tempers both likelihood and prior, and the other method tempers only the likelihood. The latter has no problem of gradient stuck during iteration.This paper discusses gradient-based tempering. As similar to a regular optimization problem, if the tempering is too slow, the algorithm will converge too fast and never reach the global optimum. If we temper too fast, on the other hand, we may skip the global optimum. Since it is difficult to find the proper schedule, we put our emphasis on learning data sequence adaptively. We use variational tempering, a method learning temperature schedule. In variational tempering, we introduce temperature as an auxiliary variable. Therefore, if the temperature is1, it will degenerate to the original variational inference. We consider finite discrete temperature, for example, 1=T1≤T2≤⋯≤TM, which allows us to compute partition function beforehand.


## 3.2. Objective Function

It is easy to derive thatpL,z,q,d,y=pL,z,q,d∣ypy. Given py=∏m=1Mπmym, the tempered joint of task difficulty, worker reliability, ground truth and worker labels can be written as
(11)pL,z,q,d∣y=pq,dcTy∏i=1NpLi,zi∣q,d1/Tyof whichTy is a temperature related to y. Thus, the model becomes
(12)pL,z,q,d,y=pq,d∏m=1MπmcTm∏i=1NpLi,zi∣q,d1/Tmym.The objective of VTI is to maximize the above joint probability. By further using the chain rule of Bayesian theory, we obtain(13)pz,q,d,y∣ψ,λ,γ=qz∣ψqq,d∣λqy∣γof which r denotes the variational parameter for the temperature. Therefore, we obtain the tempered ELBO of VTI.
(14)LTλ,ϕ;T=Eqlogpq+Eqlogpd+Eqlogpy−Eqlogqq−Eqlogqd−Eq1Ty∑i=1NEqlogpLi,zi∣q,d−EqlogCTy−∑i=1NEqlogqzi−Eqlogqy.Therefore, we can maximize this tempered ELBO, instead of optimizing the objective function directly, to obtain ground truthz and hidden parameters q,d.


## 3.3. Local Temperature

In real practice, often the data would arrive one at a time, or sometimes it is difficult to compute a huge amount of data point at once, so we use the incremental method, which computes each data point as soon as it arrives. This method can simplify the computation and show the influence of each data point to the whole network. Therefore, we can treat the potential or log value of each data point as the local factor. And instead of computing the global temperature at once, we can calculate their local temperature in a similar way. The algorithm is also more scalable to future data. Just like heat flows through the network with thermal conductivity, we can treat the potential or temperature of each local data that passes through the data network. Usingti to denote the local temperature, the joint probability of task labels, ground truth, hyperparameter, and temperature becomes
(15)pL,z,q,d,t∝pq,d∏i=1NpLi,zi∣q,d1/tipti.We can see from the above equation that adding the temperature parameter reduces the weight of the global hidden value and data, i.e., the weight of the ground truth and the task labels. In this way, the resulted distribution of the tasks will have higher entropy. Note that we no longer have to calculate the partition function since there is a∝ in the equation. By adjusting the local temperature, we can obtain a more desired outcome. For instance, outliers can be best demonstrated by higher temperature, while lower temperature can give us the main situation of the data.


## 4. VTI Algorithms

In this section, we introduce the main part of VTI. We assume that the prior distributions, i.e., the worker prior and the task prior, and the distribution of parameters of the task prior and the worker prior are all from the exponential family [21], which has the following formulation.
(16)pq,d∣θ=hq,dexpθTtq,d−agθ,pzi,Li∣q,d=hzi,Liexpq,dTtzi,Li−alq,d,of which tq,d and tzi,Li are sufficient statistics of global and local data points, of which agθ and alq,d are corresponding log normalizers.Treating the objective function as a function of the global variational parameter, we obtain(17)LTλ,ϕ;T=LTλ,ϕλ,T;T,ϕλ,T=argmaxϕijLTλ,ϕij;T.According to Hoffman, the tempered ELBO is(18)LTλ,ϕ;T=Eq1TyEqηgL,z,θT∇λagλ−λT∇λagλ+agλ−EqlogCTy−Eqlogqy.


### 4.1. Take-Home Equation

According to Hoffman, the tempered ELBO’s natural gradient with respect to the global variational parameter is(19)∇λϕλ,T=EqηgL,z,θ−λ,λ^=EqηgL,z,θ,of which ηgL,z,θ=α1+∑i=1Ntzi,Li,β1+N.
(20)λt+1=λt+ρλ^−λt.In the first step, we estimateλ^ and then update it with λt with decreasing learning rate ρt. By dividing the expectation of the sufficient statistics, we can reduce the variance of the gradients.We then optimize tempered ELBO. And the local variational update, i.e., the main body of VTI, is(21)ϕij=1TEqηlzi,−j,Li,q,d,of which, ηl is the local variational parameter and is also the natural parameter of the exponential family distribution pzij∣Li,zi,−j,q,d. Therefore, we can compute ηl by constructing the exponential family distribution pzij∣Li,zi,−j,q,d, since Li,zi,−j is known, and q,d is initialized.Therefore, VTI can be summarized in Algorithm1.Algorithm 1. VTI.
Initializeλ0,T,q0,d0,L,ϕij,ρ0Output: z, q, dfor each labelLij doUpdate local variational parameter:ϕij=1/TEqηlzi,−j,Li,q,d.end forUpdate global variational parameter:λ^=EqηgL,z,θλt+1=λt+ρλ^−λtuntil convergenceOutput:qj=probLij=zi,i∈Njdi=probLij=zi,j∈Mizi=signqjLij,j∈Mi


### 4.2. Analysis

In this part, we analyze the performance guarantee of VTI. To allocate the tasks to different workers, we use a bipartite graph algorithm, of which the left degree of the graphl refers to how many workers are needed for one task, and the right degree of the graph r refers to how many tasks are allocated to one worker at one time period. Intuitively, the error rate e is dependent on l,r,q, and d. By the definition of qj, we can see that the effective information provided by worker j can be measured by gj=∣2qj−1∣; therefore, the expectation of effective information provided by one worker is g1=E∣2qj−1∣. Let l^=l−1 and r^=r−1, u1=E2qj−1, u2=E2di−1, u=u1/u2, g2=E2di−1, and g=g1/g2. Define ρk2≡2g/u2g2l∧r∧k−1+3+1/gl^r^1−1/g2l∧r∧k−1/1−1/g2l^r^.Forg2l^r^>1, let ρ∞2≡limk→∞ρk2 such that
(22)ρ∞2=3+1/gl^r^g2l^r^g2l^r^−1.We can then arrive at the following error bound.Theorem 1.
Supposen tasks are distributed to m workers, forming a l,r bipartite graph. If the worker reliability distribution and task difficulty distribution satisfy u1>0,u2>0, and g>1/l^r^, the k-th estimates of VTI can achieve
(23)limsupm→∞1m∑i=1mℙzi≠z^iLiji,j∈E≥e−lg/2σk2.Theorem 2.
According to Theorem1, we assume n tasks are distributed to m workers, forming a l,r bipartite graph. If the worker reliability distribution and task difficulty distribution satisfy u1>0,u2>0, and g>1/l^r^, the k-th estimates of VTI can achieve
(24)limsupk→∞limsupm→∞1m∑i=1mℙzi≠z^iLiji,j∈E≥e−lg/2ρ∞2.
The implication of this theorem is that the error rate is upper bounded bye−lg/2σ∞2. Notice that the temperature parameter only affects the convergence and has no effect on error bound. We present the proof of Theorem 1 in Appendix.


### 4.3. Discussion

We developed a variational inference algorithm with tempering, which is scalable, and can analyze large amount of data by complex probabilistic graphical models. The main idea is to optimize the variational objective using stochastic optimization. By repeatedly sampling the data, it can estimate the natural gradient with noise. By introducing tempering into the objective function, we can simplify the computation complexity and improve convergence rate of the iteration. With VTI, we can easily apply label aggregation modeling to various kinds of pattern recognition and classification problem. Furthermore, VTI can be generalized to more settings as it opens the door for several research directions. Here, we analyze VTI’s complexity and how to interpret its results.First, VTI has the same runtime with majority voting, which isOml.Second, the assumption of conjugate priors is necessary. Conjugate prior plays an important role in variational inference. It enables the form of the posterior probability to resemble the prior probability, which simplifies Bayesian analysis to a large extent. Taking VTI for example, the worker and task prior are Gaussian, and their posteriors are also Gaussian. Using conjugate prior assumption, the inference model can be easily extended to different kinds of distribution that belong to the exponential family.Third, the mean-field assumption is also necessary. By approximating the true parameters of the prior distribution with a distribution that is not restricted, we can reduce computation and solve complex probabilistic graphical models.Fourth, our VTI can establish good performance under different distributions ofqj and di. Furthermore, VTI is robust to different initialization, i.e., almost every starting point can result in a unique solution. The reason for this robustness is that VTI computes the energy of the objective function and will surely decrease with each iteration.


## 4.1. Take-Home Equation

According to Hoffman, the tempered ELBO’s natural gradient with respect to the global variational parameter is(19)∇λϕλ,T=EqηgL,z,θ−λ,λ^=EqηgL,z,θ,of which ηgL,z,θ=α1+∑i=1Ntzi,Li,β1+N.
(20)λt+1=λt+ρλ^−λt.In the first step, we estimateλ^ and then update it with λt with decreasing learning rate ρt. By dividing the expectation of the sufficient statistics, we can reduce the variance of the gradients.We then optimize tempered ELBO. And the local variational update, i.e., the main body of VTI, is(21)ϕij=1TEqηlzi,−j,Li,q,d,of which, ηl is the local variational parameter and is also the natural parameter of the exponential family distribution pzij∣Li,zi,−j,q,d. Therefore, we can compute ηl by constructing the exponential family distribution pzij∣Li,zi,−j,q,d, since Li,zi,−j is known, and q,d is initialized.Therefore, VTI can be summarized in Algorithm1.Algorithm 1. VTI.
Initializeλ0,T,q0,d0,L,ϕij,ρ0Output: z, q, dfor each labelLij doUpdate local variational parameter:ϕij=1/TEqηlzi,−j,Li,q,d.end forUpdate global variational parameter:λ^=EqηgL,z,θλt+1=λt+ρλ^−λtuntil convergenceOutput:qj=probLij=zi,i∈Njdi=probLij=zi,j∈Mizi=signqjLij,j∈Mi


## 4.2. Analysis

In this part, we analyze the performance guarantee of VTI. To allocate the tasks to different workers, we use a bipartite graph algorithm, of which the left degree of the graphl refers to how many workers are needed for one task, and the right degree of the graph r refers to how many tasks are allocated to one worker at one time period. Intuitively, the error rate e is dependent on l,r,q, and d. By the definition of qj, we can see that the effective information provided by worker j can be measured by gj=∣2qj−1∣; therefore, the expectation of effective information provided by one worker is g1=E∣2qj−1∣. Let l^=l−1 and r^=r−1, u1=E2qj−1, u2=E2di−1, u=u1/u2, g2=E2di−1, and g=g1/g2. Define ρk2≡2g/u2g2l∧r∧k−1+3+1/gl^r^1−1/g2l∧r∧k−1/1−1/g2l^r^.Forg2l^r^>1, let ρ∞2≡limk→∞ρk2 such that
(22)ρ∞2=3+1/gl^r^g2l^r^g2l^r^−1.We can then arrive at the following error bound.Theorem 1.
Supposen tasks are distributed to m workers, forming a l,r bipartite graph. If the worker reliability distribution and task difficulty distribution satisfy u1>0,u2>0, and g>1/l^r^, the k-th estimates of VTI can achieve
(23)limsupm→∞1m∑i=1mℙzi≠z^iLiji,j∈E≥e−lg/2σk2.Theorem 2.
According to Theorem1, we assume n tasks are distributed to m workers, forming a l,r bipartite graph. If the worker reliability distribution and task difficulty distribution satisfy u1>0,u2>0, and g>1/l^r^, the k-th estimates of VTI can achieve
(24)limsupk→∞limsupm→∞1m∑i=1mℙzi≠z^iLiji,j∈E≥e−lg/2ρ∞2.
The implication of this theorem is that the error rate is upper bounded bye−lg/2σ∞2. Notice that the temperature parameter only affects the convergence and has no effect on error bound. We present the proof of Theorem 1 in Appendix.


## 4.3. Discussion

We developed a variational inference algorithm with tempering, which is scalable, and can analyze large amount of data by complex probabilistic graphical models. The main idea is to optimize the variational objective using stochastic optimization. By repeatedly sampling the data, it can estimate the natural gradient with noise. By introducing tempering into the objective function, we can simplify the computation complexity and improve convergence rate of the iteration. With VTI, we can easily apply label aggregation modeling to various kinds of pattern recognition and classification problem. Furthermore, VTI can be generalized to more settings as it opens the door for several research directions. Here, we analyze VTI’s complexity and how to interpret its results.First, VTI has the same runtime with majority voting, which isOml.Second, the assumption of conjugate priors is necessary. Conjugate prior plays an important role in variational inference. It enables the form of the posterior probability to resemble the prior probability, which simplifies Bayesian analysis to a large extent. Taking VTI for example, the worker and task prior are Gaussian, and their posteriors are also Gaussian. Using conjugate prior assumption, the inference model can be easily extended to different kinds of distribution that belong to the exponential family.Third, the mean-field assumption is also necessary. By approximating the true parameters of the prior distribution with a distribution that is not restricted, we can reduce computation and solve complex probabilistic graphical models.Fourth, our VTI can establish good performance under different distributions ofqj and di. Furthermore, VTI is robust to different initialization, i.e., almost every starting point can result in a unique solution. The reason for this robustness is that VTI computes the energy of the objective function and will surely decrease with each iteration.


## 5. Experiments and Results

In this section, we will assess seven advancing algorithms on real datasets for the experiments of our algorithm. First, the experimental setup is introduced, and then, the convergence performance of the collected crowdsourcing inference algorithm is analyzed. Finally, we make comparison of the error rate of VTI with the existing algorithms.


### 5.1. Experimental Setup

We use Amazon Mechanical Turk datasets (Data:https://github.com/musegoduci/variational-inference-for-crowdsourcing) to show the comparative results of the inference algorithms. The Amazon Mechanical Turk is a binary dataset, of which 0 means a worker identifies no target in a task, and 1 means the worker identifies target in a task. Note that the data need to be centralized before processing. We use bipartite assignment graph algorithm to generate the initial answer matrix for the simulation. Some notations are listed in Table 1. It should be noted that KOS (Karger, Oh, and Shah’s iterative learning algorithm for crowdsourcing inference [29]) is a message passing-based algorithm, and we first identify the reliability of each worker by some data sample and then implement MLE to estimate the test sample’s ground truth. However, BP, VMP, NMP (Liu et al.’s novel crowdsourcing inference method [22]), and VTI assume a probability distribution of worker prior and task prior and then solve the hyperparameters of the distribution, leading to a MLE estimation of the ground truth.Table 1
Notations and settings.
ParameterCommentateValuelDegree of tasks5γDegree of workers5NNumber of tasks1000MNumber of workers1000LijWorkerj’s label toward task iqjWorkerj’s reliabilitydiTaski’s difficultyziTaski’s ground truthθ1Hyperparameter of worker reliability’s distributionθ2Hyperparameter of task difficulty’s distributionα1,β1Worker prior[0.5,1]α2,β2Task prior[0.4,0.6]εConvergence tolerance10−6


### 5.2. Convergence

In this part, we analyze the convergence behavior of VTI. Note that the overall executing time for an inference algorithm depends on how long it takes to iterate once, as well as on how many times it will iterate. Since the normalizing constants can be precomputed in advance, the executing time of one iterate depends only on the calculation of the natural gradient, which gives faster convergence than others. Therefore, VTI can converge in seconds. Furthermore, we use subsampling to obtain the natural gradient of the data, which is computationally economical in each iteration. Therefore, VTI possesses advantageous computational performance.From Figure2, we can show that our VTI has a faster converge rate than KOS, MV, EM, and VMP. It is straight forward to know that this phenomenon is inherent. The reason is that VTI computes both the task difficulty and worker reliability simultaneously and reduces error in both directions. Furthermore, how fast KOS can converge depends on whether and how the ground truth is centered. If the degree of the bipartite graph is low, BP may not even converge. VMP and NMP all have rather good convergence performance because they, like VTI, are both based on the mean-field variational method.Figure 2
Convergence rate comparison.


### 5.3. Error Rate

In this research, we make comparison of VTI with MV, EM, KOS, VMP, and NMP.In Figure3, we simulate the performance of VTI with fixed right degree. Intuitively, the log error rate is positively correlated to the left degree, i.e., the quantity of tasks each worker takes. From the figure, we find that VTI has better performance under these settings.Figure 3
Comparison with fixed number of tasks per worker.In Figure4, we simulate the performance of VTI with fixed left degree. Intuitively, the log error rate is positively correlated to the right degree, i.e., the quantity of workers each task takes. From the figure, we find that VTI has better performance under these settings. In Figure 5, we simulate the performance of VTI with fixed left and right degrees. Intuitively, the log error rate is positively correlated to both the degrees. From the figure, we find that VTI has better performance under these settings.Figure 4
Comparison with fixed number of workers per task.Figure 5
Comparison with the same worker and task degree.To sum up, it can be seen from Figures3–5 that our VTI can achieve a slightly higher error rate, which proves that the inference algorithm often performs better when considering both the reliability of workers and task difficulty. And this algorithm can also ensure finding a local optimum.


## 5.1. Experimental Setup

We use Amazon Mechanical Turk datasets (Data:https://github.com/musegoduci/variational-inference-for-crowdsourcing) to show the comparative results of the inference algorithms. The Amazon Mechanical Turk is a binary dataset, of which 0 means a worker identifies no target in a task, and 1 means the worker identifies target in a task. Note that the data need to be centralized before processing. We use bipartite assignment graph algorithm to generate the initial answer matrix for the simulation. Some notations are listed in Table 1. It should be noted that KOS (Karger, Oh, and Shah’s iterative learning algorithm for crowdsourcing inference [29]) is a message passing-based algorithm, and we first identify the reliability of each worker by some data sample and then implement MLE to estimate the test sample’s ground truth. However, BP, VMP, NMP (Liu et al.’s novel crowdsourcing inference method [22]), and VTI assume a probability distribution of worker prior and task prior and then solve the hyperparameters of the distribution, leading to a MLE estimation of the ground truth.Table 1
Notations and settings.
ParameterCommentateValuelDegree of tasks5γDegree of workers5NNumber of tasks1000MNumber of workers1000LijWorkerj’s label toward task iqjWorkerj’s reliabilitydiTaski’s difficultyziTaski’s ground truthθ1Hyperparameter of worker reliability’s distributionθ2Hyperparameter of task difficulty’s distributionα1,β1Worker prior[0.5,1]α2,β2Task prior[0.4,0.6]εConvergence tolerance10−6


## 5.2. Convergence

In this part, we analyze the convergence behavior of VTI. Note that the overall executing time for an inference algorithm depends on how long it takes to iterate once, as well as on how many times it will iterate. Since the normalizing constants can be precomputed in advance, the executing time of one iterate depends only on the calculation of the natural gradient, which gives faster convergence than others. Therefore, VTI can converge in seconds. Furthermore, we use subsampling to obtain the natural gradient of the data, which is computationally economical in each iteration. Therefore, VTI possesses advantageous computational performance.From Figure2, we can show that our VTI has a faster converge rate than KOS, MV, EM, and VMP. It is straight forward to know that this phenomenon is inherent. The reason is that VTI computes both the task difficulty and worker reliability simultaneously and reduces error in both directions. Furthermore, how fast KOS can converge depends on whether and how the ground truth is centered. If the degree of the bipartite graph is low, BP may not even converge. VMP and NMP all have rather good convergence performance because they, like VTI, are both based on the mean-field variational method.Figure 2
Convergence rate comparison.


## 5.3. Error Rate

In this research, we make comparison of VTI with MV, EM, KOS, VMP, and NMP.In Figure3, we simulate the performance of VTI with fixed right degree. Intuitively, the log error rate is positively correlated to the left degree, i.e., the quantity of tasks each worker takes. From the figure, we find that VTI has better performance under these settings.Figure 3
Comparison with fixed number of tasks per worker.In Figure4, we simulate the performance of VTI with fixed left degree. Intuitively, the log error rate is positively correlated to the right degree, i.e., the quantity of workers each task takes. From the figure, we find that VTI has better performance under these settings. In Figure 5, we simulate the performance of VTI with fixed left and right degrees. Intuitively, the log error rate is positively correlated to both the degrees. From the figure, we find that VTI has better performance under these settings.Figure 4
Comparison with fixed number of workers per task.Figure 5
Comparison with the same worker and task degree.To sum up, it can be seen from Figures3–5 that our VTI can achieve a slightly higher error rate, which proves that the inference algorithm often performs better when considering both the reliability of workers and task difficulty. And this algorithm can also ensure finding a local optimum.


## 6. Conclusion

In this paper, we solve the problem of cyber physical system crowdsourcing inference using variational inference with tempering. Our VTI considers not only the worker reliability, but also the task difficulty, which makes VTI adaptable to more complex probabilistic graphical model. With the tempering procedure, the iteration process can reach a smoother local optimum and better demonstrate the relationship of balancing the influence of single data point and global variational parameter. The results are promising in reality and insightful in the field of statistical inference. Therefore, the paper contributes to the research of finding efficient algorithms in CPSs. Promising directions in future include formulating the inference problem in a more general background and exploring the connection of variational inference with all other inference and estimation algorithms.


## 7. Limitations of VTI

VTI suffers from a few limitations just like other variational inference algorithms. First, VTI relies on the assumption of the worker prior and difficulty level prior. It is applicable only to CCEF model. Second, VTI assumes that each worker has the same reliability towards different kinds of questions. Although it is capable of dealing with cases when different tasks are categorized into different classes, VTI can become extremely complex doing so. Third, VTI has the same cold start issue with other variational inference algorithms. In this paper, we initialize worker reliability and task difficulty uniformly and compute it batch by batch. However, it is also difficult to achieve a computable bipartite graph if in practice, the answers of different tasks arrive at different rate.



---

*Source: 6618980-2021-02-19.xml*