# Iterative Learning for K-Approval Votes in Crowdsourcing Systems
## 2 Related Work  
Recently, there have been some studies about \(K\)-approval votes in the crowdsourcing field. These works aim to elicit the mode of a worker's belief and propose a new paradigm of amassing high-quality responses. Specifically, in [21], the authors endeavored to obtain higher-quality labels with a new incentive mechanism that encourages good workers with additional payments. In addition, [22] proved that simple majority voting is sub-optimal,  
Figure 1: A task in the Amazon Mechanical Turk, whose goal is to categorize the item in the picture. The worker is allowed to choose two candidates.  
Figure 2: A task being distributed on Zooniverse to correctly figure out what animals Wisconsin wildlife are pictured. The worker is allowed to choose multiple candidates.  
and they brought up a conversation topic of the necessity of a probabilistic inference algorithm that can be applied to \(K\)-approval voting.  
In single-choice voting, there have been various approaches to obtaining reliable results from unreliable responses. The simplest one is majority voting. However, it is insufficient for obtaining reliable results since it regards the expertise of each worker as equal and gives the same weight to every worker. Typically, the levels of expertise are very different from experts to novices, free money collectors, and even adversarial workers [5]. In order to exploit differences of expertise among workers, Expectation and Maximization (EM) algorithms have been suggested with latent variables and unknown model parameters [9; 10; 11; 12; 23; 24].  
Alternative approaches of single voting for the binary questions have been suggested in [15] in the context of spectral methods that use low-rank matrix approximations. Additionally, the authors proposed a novel iterative learning algorithm [14; 15; 25]. Although they did not assume any prior knowledge, [PERSON] et al. [16] showed that choosing a suitable prior can improve performance via a Bayesian approach. Lately, the authors of [18; 26] proposed an iterative learning algorithm for single voting on multiple-choice questions and real-valued vector regressions, respectively. However, their algorithm cannot be applied to \(K\)-approval voting.  
In recent years, there have been some studies that built models that combine human and computer vision models. The authors of [27] proposed an online collaboration crowdsourcing system that integrates crowdsourcing platforms and computer vision machines with a Bayesian predictive model. The authors of [28; 29] developed a combined systems using confidence modeling and applied these systems to various applications in binary and multiclass setting. For efficient matching between tasks and workers, the authors of [30] proposed a bibliometric analysis on task recommendations in crowdsourcing systems (TRCS), which help workers find their preferred tasks according to their abilities. Additionally, in more general task assignment, the authors of [31] proposed a new parallel allocation of delay-tolerant tasks in practical crowdsourcing systems. As an example of a crowdsourced dataset, the authors of [32] conducted sentiment analysis experiments on a product reviews dataset [33] in the Persian language.  
There are several approaches to crowdsourcing systems via evolutionary algorithms. The authors of [34] modeled trustworthy worker selection as a multi-objective combinatorial optimization problem and solved the problem using evolutionary algorithms. Similarly, the authors of [35] developed a worker selection method with multiple objects including bug detection and cost minimization. Recently, the authors of [36] suggested a generic algorithm that uses video games as a way to gather novel solutions to optimization problems, and the authors of [37] introduced an expertise estimation as a meta-heuristic optimization harmony search problem.