The results obtained clearly show the improved performance of MA-sLDAr over the other methods. The proposed multi-annotator regression model (MA-sLDAr) was also validated with real annotators by using AMT. For that purpose, the movie review dataset from  was used. This dataset consists of 5006 movie reviews along with their respective star rating (from 1 to 10). The goal of this experiment is then predict how much a person liked a movie based on what she says about it. We ask workers to guess how much they think the writer of the review liked the movie based on her comments. An average of 4.96 answers per-review was collected for a total of 1500 reviews. The remaining reviews were used for testing. In average, each worker rated approximately 55 reviews. Using the mean answer as an estimate of the true rating of the movie yields a INLINEFORM0 of 0.830. Table TABREF99 gives an overview of the statistics of this data. Fig.

FIGREF104 shows boxplots of the number of answers per worker, as well as boxplots of their respective biases ( INLINEFORM1 ) and variances (inverse precisions, INLINEFORM2 ). The preprocessing of the text consisted of stemming and stop-words removal. Using the preprocessed data, the proposed MA-sLDAr model was compared with the same baselines that were used with the we8there dataset in Section UID98 . Fig. FIGREF105 shows the results obtained for different numbers of topics. These results show that the proposed model outperforms all the other baselines. With the purpose of verifying that the proposed model is indeed estimating the biases and precisions of the different workers correctly, we plotted the true values against the estimates of MA-sLDAr with 60 topics for a random subset of 10 workers. Fig. FIGREF106 shows the obtained results, where higher color intensities indicate higher values. Ideally, the colour of two horizontally-adjacent squares would then be of similar shades, and this is indeed what happens in practice for the majority of the workers, as Fig. FIGREF106 shows.

In order to validate the proposed classification model in real crowdsourcing settings, Amazon Mechanical Turk (AMT) was used to obtain labels from multiple annotators for two popular datasets: Reuters-21578  and LabelMe  . The Reuters-21578 is a collection of manually categorized newswire stories with labels such as Acquisitions, Crude-oil, Earnings or Grain. For this experiment, only the documents belonging to the ModApte split were considered with the additional constraint that the documents should have no more than one label. This resulted in a total of 7016 documents distributed among 8 classes. Of these, 1800 documents were submitted to AMT for multiple annotators to label, giving an average of approximately 3 answers per document (see Table TABREF81 for further details). The remaining 5216 documents were used for testing. The collected answers yield an average worker accuracy of 56.8%. Applying majority voting to these answers reveals a ground truth accuracy of 71.0%. Fig. FIGREF90 shows the boxplots of the number of answers per worker and their accuracies. Observe how applying majority voting yields a higher accuracy than the median accuracy of the workers.

The same approach was used to optimize the hyper-parameters of the all the baselines. For the svi algorithm, different mini-batch sizes and forgetting rates INLINEFORM4 were tested. For the 20-Newsgroup dataset, the best results were obtained with a mini-batch size of 500 and INLINEFORM5 . The INLINEFORM6 was kept at 1. The results are shown in Fig. FIGREF87 for different numbers of topics, where we can see that the proposed model outperforms all the baselines, being the svi version the one that performs best. In order to assess the computational advantages of the stochastic variational inference (svi) over the batch algorithm, the log marginal likelihood (or log evidence) was plotted against the number of iterations. Fig. FIGREF88 shows this comparison. Not surprisingly, the svi version converges much faster to higher values of the log marginal likelihood when compared to the batch version, which reflects the efficiency of the svi algorithm.

The results obtained by the different approaches are given in Fig. FIGREF91 , where it can be seen that the proposed model (MA-sLDAc) outperforms all the other approaches. For this dataset, the svi algorithm is using mini-batches of 300 documents. The proposed model was also validated using a dataset from the computer vision domain: LabelMe  . In contrast to the Reuters and Newsgroups corpora, LabelMe is an open online tool to annotate images. Hence, this experiment allows us to see how the proposed model generalizes beyond non-textual data. Using the Matlab interface provided in the projects' website, we extracted a subset of the LabelMe data, consisting of all the 256 x 256 images with the categories: “highway", “inside city", “tall building", “street", “forest", “coast", “mountain" or “open country". This allowed us to collect a total of 2688 labeled images. Of these, 1000 images were given to AMT workers to classify with one of the classes above.

Table TABREF99 gives an overview on the statistics of datasets used in the regression experiments. We compare the proposed model (MA-sLDAr) with the two following baselines: [itemsep=0.02cm] LDA + LinReg (mean): This baseline corresponds to applying unsupervised LDA to the data, and learning a linear regression model on the inferred topics distributions of the documents. The answers from the different annotators were aggregated by computing the mean. sLDA (mean): This corresponds to using the regression version of sLDA  with the target variables obtained by computing the mean of the annotators' answers. Fig. FIGREF102 shows the results obtained for different numbers of topics. Do to the stochastic nature of both the annotators simulation procedure and the initialization of the variational Bayesian EM algorithm, we repeated each experiment 30 times and report the average INLINEFORM0 obtained with the corresponding standard deviation. Since the regression datasets that are considered in this article are not large enough to justify the use of a stochastic variational inference (svi) algorithm, we only made experiments using the batch algorithm developed in Section SECREF61 .

Regression	As for proposed classification model, we start by validating MA-sLDAr using simulated annotators on a popular corpus where the documents have associated targets that we wish to predict. For this purpose, we shall consider a dataset of user-submitted restaurant reviews from the website we8there.com. This dataset was originally introduced in  and it consists of 6260 reviews. For each review, there is a five-star rating on four specific aspects of quality (food, service, value, and atmosphere) as well as the overall experience. Our goal is then to predict the overall experience of the user based on his comments in the review. We apply the same preprocessing as in  , which consists in tokenizing the text into bigrams and discarding those that appear in less than ten reviews. The preprocessing of the documents consisted of stemming and stop-words removal. After that, 75% of the documents were randomly selected for training and the remaining 25% for testing. As with the classification model, we seek to simulate an heterogeneous set of annotators in terms of reliability and bias.

Then, the answers are sampled i.i.d. from INLINEFORM6 . This procedure was used to simulate 5 different annotators with the following accuracies: 0.737, 0.468, 0.284, 0.278, 0.260. In this experiment, no repeated labelling was used. Hence, each annotator only labels roughly one-fifth of the data. When compared to the ground truth, the simulated answers revealed an accuracy of 0.405. See Table TABREF81 for an overview of the details of the classification datasets used. Both the batch and the stochastic variational inference (svi) versions of the proposed model (MA-sLDAc) are compared with the following baselines: [itemsep=0.02cm] LDA + LogReg (mv): This baseline corresponds to applying unsupervised LDA to the data, and learning a logistic regression classifier on the inferred topics distributions of the documents. The labels from the different annotators were aggregated using majority voting (mv). Notice that, when there is a single annotator label per instance, majority voting is equivalent to using that label for training.

Classification	In order to first validate the proposed model for classification problems in a slightly more controlled environment, the well-known 20-Newsgroups benchmark corpus  was used by simulating multiple annotators with different levels of expertise. The 20-Newsgroups consists of twenty thousand messages taken from twenty newsgroups, and is divided in six super-classes, which are, in turn, partitioned in several sub-classes. For this first set of experiments, only the four most populated super-classes were used: “computers", “science", “politics" and “recreative". The preprocessing of the documents consisted of stemming and stop-words removal. After that, 75% of the documents were randomly selected for training and the remaining 25% for testing. The different annotators were simulated by sampling their answers from a multinomial distribution, where the parameters are given by the lines of the annotators' confusion matrices. Hence, for each annotator INLINEFORM0 , we start by pre-defining a confusion matrix INLINEFORM1 with elements INLINEFORM2 , which correspond to the probability that the annotators' answer is INLINEFORM3 given that the true label is INLINEFORM4 , INLINEFORM5 .

This is the case of the 20-Newsgroups' simulated annotators, but the same does not apply for the experiments in Section UID89 . LDA + Raykar: For this baseline, the model of  was applied using the documents' topic distributions inferred by LDA as features. LDA + Rodrigues: This baseline is similar to the previous one, but uses the model of  instead. Blei 2003 (mv): The idea of this baseline is to replicate a popular state-of-the-art approach for document classification. Hence, the approach of  was used. It consists of applying LDA to extract the documents' topics distributions, which are then used to train a SVM. Similarly to the previous approach, the labels from the different annotators were aggregated using majority voting (mv). sLDA (mv): This corresponds to using the classification version of sLDA  with the labels obtained by performing majority voting (mv) on the annotators' answers. For all the experiments the hyper-parameters INLINEFORM0 , INLINEFORM1 and INLINEFORM2 were set using a simple grid search in the collection INLINEFORM3 .

Hence, in order to simulate an annotator INLINEFORM0 , we proceed as follows: let INLINEFORM1 be the true review of the restaurant; we start by assigning a given bias INLINEFORM2 and precision INLINEFORM3 to the reviewers, depending on what type of annotator we wish to simulate (see Fig. FIGREF45 ); we then sample a simulated answer as INLINEFORM4 . Using this procedure, we simulated 5 annotators with the following (bias, precision) pairs: (0.1, 10), (-0.3, 3), (-2.5, 10), (0.1, 0.5) and (1, 0.25). The goal is to have 2 good annotators (low bias, high precision), 1 highly biased annotator and 2 low precision annotators where one is unbiased and the other is reasonably biased. The coefficients of determination ( INLINEFORM5 ) of the simulated annotators are: [0.940, 0.785, -2.469, -0.131, -1.749]. Computing the mean of the answers of the different annotators yields a INLINEFORM6 of 0.798.

For this, we use various popular datasets from the state-of-the-art, that are commonly used for benchmarking machine learning algorithms. Finally, an efficient stochastic variational inference algorithm was described, which gives the proposed models the ability to scale to large datasets.

Optimizing w.r.t. INLINEFORM3 gives a similar update to the one in sLDA  DISPLAYFORM0   where we defined INLINEFORM0 . Notice how this update differs only from the one in  by replacing the true target variable by its expected value under the variational distribution, which is given by INLINEFORM1 . The only variables left for doing inference on are then the latent true targets INLINEFORM0 . The variational distribution of INLINEFORM1 is governed by two parameters: a mean INLINEFORM2 and a variance INLINEFORM3 . Collecting all the terms in INLINEFORM4 that contain INLINEFORM5 gives DISPLAYFORM0   Taking derivatives of INLINEFORM0 and setting them to zero gives the following update for INLINEFORM1 DISPLAYFORM0   Notice how the value of INLINEFORM0 is a weighted average of what the linear regression model on the empirical topic mixture believes the true target should be, and the bias-corrected answers of the different annotators weighted by their individual precisions. As for INLINEFORM0 , we can optimize INLINEFORM1 w.r.t. INLINEFORM2 by collecting all terms that contain INLINEFORM3 DISPLAYFORM0   and taking derivatives, yielding the update DISPLAYFORM0

In order to verify that the proposed model was estimating the (normalized) confusion matrices INLINEFORM0 of the different workers correctly, a random sample of them was plotted against the true confusion matrices (i.e. the normalized confusion matrices evaluated against the true labels). Figure FIGREF95 shows the results obtained with 60 topics on the Reuters-21578 dataset, where the color intensity of the cells increases with the magnitude of the value of INLINEFORM1 (the supplementary material provides a similar figure for the LabelMe dataset). Using this visualization we can verify that the AMT workers are quite heterogeneous in their labeling styles and in the kind of mistakes they make, with several workers showing clear biases (e.g. workers 3 and 4), while others made mistakes more randomly (e.g. worker 1). Nevertheless, the proposed is able to capture these patterns correctly and account for effect.

We propose two different models, one for classification  and another for regression problems, thus covering a very wide range of possible practical applications, as we empirically demonstrate. Since the majority of the tasks for which multiple annotators are used generally involve complex data such as text, images and video, by developing a multi-annotator supervised topic model we are contributing with a powerful tool for learning predictive models of complex high-dimensional data from crowds. Given that the increasing sizes of modern datasets can pose a problem for obtaining human labels as well as for Bayesian inference, we propose an efficient stochastic variational inference algorithm  that is able to scale to very large datasets. We empirically show, using both simulated and real multiple-annotator labels obtained from AMT for popular text and image collections, that the proposed models are able to outperform other state-of-the-art approaches in both classification and regression tasks. We further show the computational and predictive advantages of the stochastic variational inference algorithm over its batch counterpart.

With the purpose of comparing the proposed model with a popular state-of-the-art approach for image classification, for the LabelMe dataset, the following baseline was introduced: Bosch 2006 (mv): This baseline is similar to one in  . The authors propose the use of pLSA to extract the latent topics, and the use of k-nearest neighbor (kNN) classifier using the documents' topics distributions. For this baseline, unsupervised LDA is used instead of pLSA, and the labels from the different annotators for kNN (with INLINEFORM0 ) are aggregated using majority voting (mv). The results obtained by the different approaches for the LabelMe data are shown in Fig. FIGREF94 , where the svi version is using mini-batches of 200 documents. Analyzing the results for the Reuters-21578 and LabelMe data, we can observe that MA-sLDAc outperforms all the baselines, with slightly better accuracies for the batch version, especially in the Reuters data. Interestingly, the second best results are consistently obtained by the multi-annotator approaches, which highlights the need for accounting for the noise and biases of the answers of the different annotators.

Stochastic variational inference	In Section SECREF21 , we proposed a batch coordinate ascent algorithm for doing variational inference in the proposed model. This algorithm iterates between analyzing every document in the corpus to infer the local hidden structure, and estimating the global hidden variables. However, this can be inefficient for large datasets, since it requires a full pass through the data at each iteration before updating the global variables. In this section, we develop a stochastic variational inference algorithm  , which follows noisy estimates of the gradients of the evidence lower bound INLINEFORM0 . Based on the theory of stochastic optimization  , we can find unbiased estimates of the gradients by subsampling a document (or a mini-batch of documents) from the corpus, and using it to compute the gradients as if that document was observed INLINEFORM0 times. Hence, given an uniformly sampled document INLINEFORM1 , we use the current posterior distributions of the global latent variables, INLINEFORM2 and INLINEFORM3 , and the current coefficient estimates INLINEFORM4 , to compute the posterior distribution over the local hidden variables INLINEFORM5 , INLINEFORM6 and INLINEFORM7 using Eqs. EQREF25 , EQREF33 and EQREF29 respectively.

Introduction	Topic models, such as latent Dirichlet allocation (LDA), allow us to analyze large collections of documents by revealing their underlying themes, or topics, and how each document exhibits them  . Therefore, it is not surprising that topic models have become a standard tool in data analysis, with many applications that go even beyond their original purpose of modeling textual data, such as analyzing images  ,  , videos  , survey data  or social networks data  . Since documents are frequently associated with other variables such as labels, tags or ratings, much interest has been placed on supervised topic models  , which allow the use of that extra information to “guide" the topics discovery. By jointly learning the topics distributions and a classification or regression model, supervised topic models have been shown to outperform the separate use of their unsupervised analogues together with an external regression/classification algorithm  ,  . Supervised topics models are then state-of-the-art approaches for predicting target variables associated with complex high-dimensional data, such as documents or images. Unfortunately, the size of modern datasets makes the use of a single annotator unrealistic and unpractical for the majority of the real-world applications that involve some form of human labeling.

Conclusion	This article proposed a supervised topic model that is able to learn from multiple annotators and crowds, by accounting for their biases and different levels of expertise. Given the large sizes of modern datasets, and considering that the majority of the tasks for which crowdsourcing and multiple annotators are desirable candidates, generally involve complex high-dimensional data such as text and images, the proposed model constitutes a strong contribution for the multi-annotator paradigm. This model is then capable of jointly modeling the words in documents as arising from a mixture of topics, as well as the latent true target variables and the (noisy) answers of the multiple annotators. We developed two distinct models, one for classification and another for regression, which share similar intuitions but that inevitably differ due to the nature of the target variables. We empirically showed, using both simulated and real annotators from Amazon Mechanical Turk that the proposed model is able to outperform state-of-the-art approaches in several real-world problems, such as classifying posts, news stories and images, or predicting the number of stars of restaurant and the rating of movie based on their reviews.

Approximate inference	Given a dataset INLINEFORM0 , the goal of inference is to compute the posterior distribution of the per-document topic proportions INLINEFORM1 , the per-word topic assignments INLINEFORM2 , the per-topic distribution over words INLINEFORM3 , the per-document latent true class INLINEFORM4 , and the per-annotator confusion parameters INLINEFORM5 . As with LDA, computing the exact posterior distribution of the latent variables is computationally intractable. Hence, we employ mean-field variational inference to perform approximate Bayesian inference. Variational inference methods seek to minimize the KL divergence between the variational and the true posterior distribution. We assume a fully-factorized (mean-field) variational distribution of the form DISPLAYFORM0   where INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 and INLINEFORM4 are variational parameters. Table TABREF23 shows the correspondence between variational parameters and the original parameters. Let INLINEFORM0 denote the model parameters. Following  , the KL minimization can be equivalently formulated as maximizing the following lower bound on the log marginal likelihood DISPLAYFORM0   which we maximize using coordinate ascent. Optimizing INLINEFORM0 w.r.t.

To gain further insights, Table TABREF96 shows 4 example images from the LabelMe dataset, along with their true labels, the answers provided by the different workers, the true label inferred by the proposed model and the likelihood of the different possible answers given the true label for each annotator ( INLINEFORM0 for INLINEFORM1 ) using a color-coding scheme similar to Fig. FIGREF95 . In the first example, although majority voting suggests “inside city" to be the correct label, we can see that the model has learned that annotators 32 and 43 are very likely to provide the label “inside city" when the true label is actually “street", and it is able to leverage that fact to infer that the correct label is “street". Similarly, in the second image the model is able to infer the correct true label from 3 conflicting labels. However, in the third image the model is not able to recover the correct true class, which can be explained by it not having enough evidence about the annotators and their reliabilities and biases (likelihood distribution for these cases is uniform). In fact, this raises interesting questions regarding requirements for the minimum number of labels per annotator, their reliabilities and their coherence.

Learning Supervised Topic Models for Classification and Regression from Crowds	The growing need to analyze large collections of documents has led to great developments in topic modeling. Since documents are frequently associated with other related variables, such as labels or ratings, much interest has been placed on supervised topic models. However, the nature of most annotation tasks, prone to ambiguity and noise, often with high volumes of documents, deem learning under a single-annotator assumption unrealistic or unpractical for most real-world applications. In this article, we propose two supervised topic models, one for classification and another for regression problems, which account for the heterogeneity and biases among different annotators that are encountered in practice when learning from crowds. We develop an efficient stochastic variational inference algorithm that is able to scale to very large datasets, and we empirically demonstrate the advantages of the proposed model over state-of-the-art approaches.

[t] Stochastic variational inference for the proposed regression model [1] Initialize INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 , INLINEFORM4 , INLINEFORM5 , INLINEFORM6 Set t = t + 1 Sample a document INLINEFORM7 uniformly from the corpus Compute INLINEFORM8 using Eq. EQREF64 , for INLINEFORM9 Compute INLINEFORM10 using Eq. EQREF25 Compute INLINEFORM11 using Eq. EQREF66 Compute INLINEFORM12 using Eq. EQREF68 local parameters INLINEFORM13 , INLINEFORM14 and INLINEFORM15 converge Compute step-size INLINEFORM16 Update topics variational parameters DISPLAYFORM0   global convergence criterion is met

These posteriors are then used to update the global variational parameters, INLINEFORM8 and INLINEFORM9 by taking a step of size INLINEFORM10 in the direction of the noisy estimates of the natural gradients. Algorithm SECREF37 describes a stochastic variational inference algorithm for the proposed model. Given an appropriate schedule for the learning rates INLINEFORM0 , such that INLINEFORM1 and INLINEFORM2 , the stochastic optimization algorithm is guaranteed to converge to a local maximum of the evidence lower bound  . [t] Stochastic variational inference for the proposed classification model [1] Initialize INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 , INLINEFORM4 , INLINEFORM5 Set t = t + 1 Sample a document INLINEFORM6 uniformly from the corpus Compute INLINEFORM7 using Eq. EQREF33 , for INLINEFORM8 Compute INLINEFORM9 using Eq. EQREF25 Compute INLINEFORM10 using Eq.

Experiments	In this section, the proposed multi-annotator supervised LDA models for classification and regression (MA-sLDAc and MA-sLDAr, respectively) are validated using both simulated annotators on popular corpora and using real multiple-annotator labels obtained from Amazon Mechanical Turk. Namely, we shall consider the following real-world problems: classifying posts and news stories; classifying images according to their content; predicting number of stars that a given user gave to a restaurant based on the review; predicting movie ratings using the text of the reviews.

Parameter estimation	The model parameters are INLINEFORM0 . The parameters INLINEFORM1 of the Dirichlet priors can be regarded as hyper-parameters of the proposed model. As with many works on topic models (e.g.  ,  ), we assume hyper-parameters to be fixed, since they can be effectively selected by grid-search procedures which are able to explore well the parameter space without suffering from local optima. Our focus is then on estimating the coefficients INLINEFORM2 using a variational EM algorithm. Therefore, in the E-step we use the variational inference algorithm from section SECREF21 to estimate the posterior distribution of the latent variables, and in the M-step we find maximum likelihood estimates of INLINEFORM3 by maximizing the evidence lower bound INLINEFORM4 . Unfortunately, taking derivatives of INLINEFORM5 w.r.t. INLINEFORM6 does not yield a closed-form solution. Hence, we use a numerical method, namely L-BFGS  , to find an optimum. The objective function and gradients are given by DISPLAYFORM0   where, for convenience, we defined the following variable: INLINEFORM0 . The parameters of the proposed regression model are INLINEFORM0 .

INLINEFORM0 gives the following estimate for the bias of the INLINEFORM1 annotator DISPLAYFORM0  Similarly, we can find maximum likelihood estimates for the precisions INLINEFORM0 of the different annotators by considering the terms in INLINEFORM1 that contain INLINEFORM2 DISPLAYFORM0   The maximum likelihood estimate for the precision (inverse variance) of the INLINEFORM0 annotator is then given by DISPLAYFORM0  Given a set of fitted parameters, it is then straightforward to make predictions for new documents: it is just necessary to infer the (approximate) posterior distribution over the word-topic assignments INLINEFORM0 for all the words using the coordinates ascent updates of standard LDA (Eqs. EQREF25 and EQREF42 ), and then use the mean topic assignments INLINEFORM1 to make predictions INLINEFORM2 .

Learning from multiple annotators	Learning from multiple annotators is an increasingly important research topic. Since the early work of Dawid and Skeene  , who attempted to obtain point estimates of the error rates of patients given repeated but conflicting responses to various medical questions, many approaches have been proposed. These usually rely on latent variable models. For example, in  the authors propose a model to estimate the ground truth from the labels of multiple experts, which is then used to train a classifier. While earlier works usually focused on estimating the ground truth and the error rates of different annotators, recent works are more focused on the problem of learning classifiers using multiple-annotator data. This idea was explored by Raykar et al.  , who proposed an approach for jointly learning the levels of expertise of different annotators and the parameters of a logistic regression classifier, by modeling the ground truth labels as latent variables. This work was later extended in  by considering the dependencies of the annotators' labels on the instances they are labeling, and also in  through the use of Gaussian process classifiers. The model proposed in this article for classification problems shares the same intuition with this line of work and models the true labels as latent variables.

The generative process of the proposed model for classification problems can then be summarized as follows: For each annotator INLINEFORM0  For each class INLINEFORM0  Draw reliability parameter INLINEFORM0  For each topic INLINEFORM0  Draw topic distribution INLINEFORM0  For each document INLINEFORM0  Draw topic proportions INLINEFORM0  For the INLINEFORM0 word Draw topic assignment INLINEFORM0  Draw word INLINEFORM0  Draw latent (true) class INLINEFORM0  For each annotator INLINEFORM0  Draw annotator's label INLINEFORM0  where INLINEFORM0 denotes the set of annotators that labeled the INLINEFORM1 document, INLINEFORM2 , and the softmax is given by DISPLAYFORM0  Fig. FIGREF20 shows a graphical model representation of the proposed model, where INLINEFORM0 denotes the number of topics, INLINEFORM1 is the number of classes, INLINEFORM2 is the total number of annotators and INLINEFORM3 is the number of words in the document INLINEFORM4 . Shaded nodes are used to distinguish latent variable from the observed ones and small solid circles are used to denote model parameters.

Each image was labeled by an average of 2.547 workers, with a mean accuracy of 69.2%. When majority voting is applied to the collected answers, a ground truth accuracy of 76.9% is obtained. Fig. FIGREF92 shows the boxplots of the number of answers per worker and their accuracies. Interestingly, the worker accuracies are much higher and their distribution is much more concentrated than on the Reuters-21578 data (see Fig. FIGREF90 ), which suggests that this is an easier task for the AMT workers. The preprocessing of the images used is similar to the approach in  . It uses 128-dimensional SIFT  region descriptors selected by a sliding grid spaced at one pixel. This sliding grid extracts local regions of the image with sizes uniformly sampled between 16 x 16 and 32 x 32 pixels. The 128-dimensional SIFT descriptors produced by the sliding window are then fed to a k-means algorithm (with k=200) in order construct a vocabulary of 200 “visual words". This allows us to represent the images with a bag of visual words model.

The Dirichlet-multimonial regression (DMR) topic model  includes a log-linear prior on the document's mixture proportions that is a function of a set of arbitrary features, such as author, date, publication venue or references in scientific articles. The inferred Dirichlet-multinomial distribution can then be used to make predictions about the values of theses features. The inverse regression topic model (IRTM)  is a mixed-membership extension of the multinomial inverse regression (MNIR) model proposed in  that exploits the topical structure of text corpora to improve its predictions and facilitate exploratory data analysis. However, this results in a rather complex and inefficient inference procedure. Furthermore, making predictions in the IRTM is not trivial. For example, MAP estimates of targets will be in a different scale than the original document's metadata. Hence, the authors propose the use of a linear model to regress metadata values onto their MAP predictions. The approaches discussed so far rely on likelihood-based estimation procedures. The work in  contrasts with these approaches by proposing MedLDA, a supervised topic model that utilizes the max-margin principle for estimation.

In this article, we take this idea one step further by also considering a per-annotator bias parameter, which gives the proposed model the ability to overcome certain personal tendencies in the annotators labeling styles that are quite common, for example, in product ratings and document reviews. Furthermore, we empirically validate the proposed model using real multi-annotator data obtained from Amazon Mechanical Turk. This contrasts with the previously mentioned works, which rely only on simulated annotators.

Classification model	In this section, we develop a multi-annotator supervised topic model for classification problems. The model for regression settings will be presented in Section SECREF5 . We start by deriving a (batch) variational inference algorithm for approximating the posterior distribution over the latent variables and an algorithm to estimate the model parameters. We then develop a stochastic variational inference algorithm that gives the model the capability of handling large collections of documents. Finally, we show how to use the learned model to classify new documents.

Unfortunately, a direct mapping between document classes and the labels provided by the different annotators in a multiple-annotator setting would correspond to assuming that they are all equally reliable, an assumption that is violated in practice, as previous works clearly demonstrate (e.g.  ,  ). Hence, we assume the existence of a latent ground truth class, and model the labels from the different annotators using a noise model that states that, given a true class INLINEFORM0 , each annotator INLINEFORM1 provides the label INLINEFORM2 with some probability INLINEFORM3 . Hence, by modeling the matrix INLINEFORM4 we are in fact modeling a per-annotator (normalized) confusion matrix, which allows us to account for their different levels of expertise and correct their potential biases.

As we did for the classification model, we shall assume the Dirichlet parameters, INLINEFORM1 and INLINEFORM2 , to be fixed. Similarly, we shall assume that the variance of the true targets, INLINEFORM3 , to be constant. The only parameters left to estimate are then the regression coefficients INLINEFORM4 and the annotators biases, INLINEFORM5 , and precisions, INLINEFORM6 , which we estimate using variational Bayesian EM. Since the latent true targets are now linear functions of the documents' empirical topic mixtures (i.e. there is no softmax function), we can find a closed form solution for the regression coefficients INLINEFORM0 . Taking derivatives of INLINEFORM1 w.r.t. INLINEFORM2 and setting them to zero, gives the following solution for INLINEFORM3 DISPLAYFORM0   where DISPLAYFORM0  We can find maximum likelihood estimates for the annotator biases INLINEFORM0 by optimizing the lower bound on the marginal likelihood. The terms in INLINEFORM1 that involve INLINEFORM2 are DISPLAYFORM0   Taking derivatives w.r.t.

Document classification	In order to make predictions for a new (unlabeled) document INLINEFORM0 , we start by computing the approximate posterior distribution over the latent variables INLINEFORM1 and INLINEFORM2 . This can be achieved by dropping the terms that involve INLINEFORM3 , INLINEFORM4 and INLINEFORM5 from the model's joint distribution (since, at prediction time, the multi-annotator labels are no longer observed) and averaging over the estimated topics distributions. Letting the topics distribution over words inferred during training be INLINEFORM6 , the joint distribution for a single document is now simply given by DISPLAYFORM0   Deriving a mean-field variational inference algorithm for computing the posterior over INLINEFORM0 results in the same fixed-point updates as in LDA  for INLINEFORM1 (Eq. EQREF25 ) and INLINEFORM2 DISPLAYFORM0   Using the inferred posteriors and the coefficients INLINEFORM0 estimated during training, we can make predictions as follows DISPLAYFORM0   This is equivalent to making predictions in the classification version of sLDA  .

However, it differs significantly by using a fully Bayesian approach for estimating the reliabilities and biases of the different annotators. Furthermore, it considers the problems of learning a low-dimensional representation of the input data (through topic modeling) and modeling the answers of multiple annotators jointly, providing an efficient stochastic variational inference algorithm. Despite the considerable amount of approaches for learning classifiers from the noisy answers of multiple annotators, for continuous response variables this problem has been approached in a much smaller extent. For example, Groot et al.  address this problem in the context of Gaussian processes. In their work, the authors assign a different variance to the likelihood of the data points provided by the different annotators, thereby allowing them to have different noise levels, which can be estimated by maximizing the marginal likelihood of the data. Similarly, the authors in  propose an extension of their own classification approach to regression problems by assigning different variances to the Gaussian noise models of the different annotators.

The main difficulty of applying standard variational inference methods to the proposed model is the non-conjugacy between the distribution of the mean topic-assignment INLINEFORM0 and the softmax. Namely, in the expectation DISPLAYFORM0   the second term is intractable to compute. We can make progress by applying Jensen's inequality to bound it as follows DISPLAYFORM0   where INLINEFORM0 , which is constant w.r.t. INLINEFORM1 . This local variational bound can be made tight by noticing that INLINEFORM2 , where equality holds if and only if INLINEFORM3 . Hence, given the current parameter estimates INLINEFORM4 , if we set INLINEFORM5 and INLINEFORM6 then, for an individual parameter INLINEFORM7 , we have that DISPLAYFORM0   Using this local bound to approximate the expectation of the log-sum-exp term, and taking derivatives of the evidence lower bound w.r.t. INLINEFORM0 with the constraint that INLINEFORM1 , yields the following fix-point update DISPLAYFORM0   where INLINEFORM0 denotes the size of the vocabulary. Notice how the per-word variational distribution over topics INLINEFORM1 depends on the variational distribution over the true class label INLINEFORM2 .

The variational inference algorithm iterates between Eqs. EQREF25 - EQREF33 until the evidence lower bound, Eq. EQREF24 , converges. Additional details are provided as supplementary material.   The goal of inference is to compute the posterior distribution of the per-document topic proportions INLINEFORM0 , the per-word topic assignments INLINEFORM1 , the per-topic distribution over words INLINEFORM2 and the per-document latent true targets INLINEFORM3 . As we did for the classification model, we shall develop a variational inference algorithm using coordinate ascent. The lower-bound on the log marginal likelihood is now given by DISPLAYFORM0   where INLINEFORM0 are the model parameters. We assume a fully-factorized (mean-field) variational distribution INLINEFORM1 of the form DISPLAYFORM0   where INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 and INLINEFORM4 are the variational parameters. Notice the new Gaussian term, INLINEFORM5 , corresponding to the approximate posterior distribution of the unobserved true targets. Optimizing the variational objective INLINEFORM0 w.r.t. INLINEFORM1 and INLINEFORM2 yields the same updates from Eqs. EQREF25 and .

Hence, the interest of researchers on building models that take the reliabilities of different annotators into consideration and mitigate the effect of their biases has spiked during the last few years (e.g.  ,  ). The increasing popularity of crowdsourcing platforms like Amazon Mechanical Turk (AMT) has further contributed to the recent advances in learning from crowds. This kind of platforms offers a fast, scalable and inexpensive solution for labeling large amounts of data. However, their heterogeneous nature in terms of contributors makes their straightforward application prone to many sorts of labeling noise and bias. Hence, a careless use of crowdsourced data as training data risks generating flawed models. In this article, we propose a fully generative supervised topic model that is able to account for the different reliabilities of multiple annotators and correct their biases. The proposed model is then capable of jointly modeling the words in documents as arising from a mixture of topics, the latent true target variables as a result of the empirical distribution over topics of the documents, and the labels of the multiple annotators as noisy versions of that latent ground truth.

Notice that we included a Dirichlet prior over the topics INLINEFORM5 to produce a smooth posterior and control sparsity. Similarly, instead of computing maximum likelihood or MAP estimates for the annotators reliability parameters INLINEFORM6 , we place a Dirichlet prior over these variables and perform approximate Bayesian inference. This contrasts with previous works on learning classification models from crowds  ,  . For developing a multi-annotator supervised topic model for regression, we shall follow a similar intuition as the one we considered for classification. Namely, we shall assume that, for a given document INLINEFORM0 , each annotator provides a noisy version, INLINEFORM1 , of the true (continuous) target variable, which we denote by INLINEFORM2 . This can be, for example, the true rating of a product or the true sentiment of a document. Assuming that each annotator INLINEFORM3 has its own personal bias INLINEFORM4 and precision INLINEFORM5 (inverse variance), and assuming a Gaussian noise model for the annotators' answers, we have that DISPLAYFORM0   This approach is therefore more powerful than previous works  ,  , where a single precision parameter was used to model the annotators' expertise. Fig.

For instance, the popular Reuters-21578 benchmark corpus was categorized by a group of personnel from Reuters Ltd and Carnegie Group, Inc. Similarly, the LabelMe project asks volunteers to annotate images from a large collection using an online tool. Hence, it is seldom the case where a single oracle labels an entire collection. Furthermore, the Web, through its social nature, also exploits the wisdom of crowds to annotate large collections of documents and images. By categorizing texts, tagging images or rating products and places, Web users are generating large volumes of labeled content. However, when learning supervised models from crowds, the quality of labels can vary significantly due to task subjectivity and differences in annotator reliability (or bias)  ,  . If we consider a sentiment analysis task, it becomes clear that the subjectiveness of the exercise is prone to generate considerably distinct labels from different annotators. Similarly, online product reviews are known to vary considerably depending on the personal biases and volatility of the reviewer's opinions. It is therefore essential to account for these issues when learning from this increasingly common type of data.

This approach is coherent with the generative perspective of LDA but, nevertheless, several discriminative alternatives also exist. For example, DiscLDA  introduces a class-dependent linear transformation on the topic mixture proportions of each document, such that the per-word topic assignments are drawn from linearly transformed mixture proportions. The class-specific transformation matrices are then able to reposition the topic mixture proportions so that documents with the same class labels have similar topics mixture proportions. The transformation matrices can be estimated by maximizing the conditional likelihood of response variables as the authors propose  . An alternative way of including classes in LDA for supervision is the one proposed in the Labeled-LDA model  . Labeled-LDA is a variant of LDA that incorporates supervision by constraining the topic model to assign to a document only topics that correspond to its label set. While this allows for multiple labels per document, it is restrictive in the sense that the number of topics needs to be the same as the number of possible labels. From a regression perspective, other than sLDA, the most relevant approaches are the Dirichlet-multimonial regression  and the inverse regression topic models  .

Despite its margin-based advantages, MedLDA looses the probabilistic interpretation of the document classes given the topic mixture distributions. On the contrary, in this article we propose a fully generative probabilistic model of the answers of multiple annotators and of the words of documents arising from a mixture of topics.

She was responsible/participated in several research projects in a wide range of application areas such as Text Classification, Financial, Biomedical and Bioinformatics. Bernardete Ribeiro is IEEE Senior Member, and member of IARP International Association of Pattern Recognition and ACM. []Francisco C. Pereira is Full Professor at the Technical University of Denmark (DTU), where he leads the Smart Mobility research group. His main research focus is on applying machine learning and pattern recognition to the context of transportation systems with the purpose of understanding and predicting mobility behavior, and modeling and optimizing the transportation system as a whole. He has Master€™s (2000) and Ph.D. (2005) degrees in Computer Science from University of Coimbra, and has authored/co-authored over 70 journal and conference papers in areas such as pattern recognition, transportation, knowledge based systems and cognitive science. Francisco was previously Research Scientist at MIT and Assistant Professor in University of Coimbra. He was awarded several prestigious prizes, including an IEEE Achievements award, in 2009, the Singapore GYSS Challenge in 2013, and the Pyke Johnson award from Transportation Research Board, in 2015.

Proposed model	Let INLINEFORM0 be an annotated corpus of size INLINEFORM1 , where each document INLINEFORM2 is given a set of labels INLINEFORM3 from INLINEFORM4 distinct annotators. We can take advantage of the inherent topical structure of documents and model their words as arising from a mixture of topics, each being defined as a distribution over the words in a vocabulary, as in LDA. In LDA, the INLINEFORM5 word, INLINEFORM6 , in a document INLINEFORM7 is provided a discrete topic-assignment INLINEFORM8 , which is drawn from the documents' distribution over topics INLINEFORM9 . This allows us to build lower-dimensional representations of documents, which we can explore to build classification models by assigning coefficients INLINEFORM10 to the mean topic-assignment of the words in the document, INLINEFORM11 , and applying a softmax function in order to obtain a distribution over classes. Alternatively, one could consider more flexible models such as Gaussian processes, however that would considerably increase the complexity of inference.

FIGREF45 illustrates this intuition for 4 annotators, represented by different colors. The “green annotator" is the best one, since he is right on the target and his answers vary very little (low bias, high precision). The “yellow annotator" has a low bias, but his answers are very uncertain, as they can vary a lot. Contrarily, the “blue annotator" is very precise, but consistently over-estimates the true target (high bias, high precision). Finally, the “red annotator" corresponds to the worst kind of annotator (with high bias and low precision). Having specified a model for annotators answers given the true targets, the only thing left is to do is to specify a model of the latent true targets INLINEFORM0 given the empirical topic mixture distributions INLINEFORM1 . For this, we shall keep things simple and assume a linear model as in sLDA  .

The generative process of the proposed model for continuous target variables can then be summarized as follows: For each annotator INLINEFORM0  For each class INLINEFORM0  Draw reliability parameter INLINEFORM0  For each topic INLINEFORM0  Draw topic distribution INLINEFORM0  For each document INLINEFORM0  Draw topic proportions INLINEFORM0  For the INLINEFORM0 word Draw topic assignment INLINEFORM0  Draw word INLINEFORM0  Draw latent (true) target INLINEFORM0  For each annotator INLINEFORM0  Draw answer INLINEFORM0  Fig. FIGREF60 shows a graphical representation of the proposed model.

INLINEFORM1 and INLINEFORM2 gives the same coordinate ascent updates as in LDA  DISPLAYFORM0  The variational Dirichlet parameters INLINEFORM0 can be optimized by collecting only the terms in INLINEFORM1 that contain INLINEFORM2 DISPLAYFORM0   where INLINEFORM0 denotes the documents labeled by the INLINEFORM1 annotator, INLINEFORM2 , and INLINEFORM3 and INLINEFORM4 are the gamma and digamma functions, respectively. Taking derivatives of INLINEFORM5 w.r.t. INLINEFORM6 and setting them to zero, yields the following update DISPLAYFORM0  Similarly, the coordinate ascent updates for the documents distribution over classes INLINEFORM0 can be found by considering the terms in INLINEFORM1 that contain INLINEFORM2 DISPLAYFORM0   where INLINEFORM0 . Adding the necessary Lagrange multipliers to ensure that INLINEFORM1 and setting the derivatives w.r.t. INLINEFORM2 to zero gives the following update DISPLAYFORM0   Observe how the variational distribution over the true classes results from a combination between the dot product of the inferred mean topic assignment INLINEFORM0 with the coefficients INLINEFORM1 and the labels INLINEFORM2 from the multiple annotators “weighted" by their expected log probability INLINEFORM3 .

Acknowledgment	The Fundação para a Ciência e Tecnologia (FCT) is gratefully acknowledged for founding this work with the grants SFRH/BD/78396/2011 and PTDC/ECM-TRA/1898/2012 (InfoCROWDS). []Mariana Lourenço has a MSc degree in Informatics Engineering from University of Coimbra, Portugal. Her thesis presented a supervised topic model that is able to learn from crowds and she took part in a research project whose primary objective was to exploit online information about public events to build predictive models of flows of people in the city. Her main research interests are machine learning, pattern recognition and natural language processing. []Bernardete Ribeiro is Associate Professor at the Informatics Engineering Department, University of Coimbra in Portugal, from where she received a D.Sc. in Informatics Engineering, a Ph.D. in Electrical Engineering, speciality of Informatics, and a MSc in Computer Science. Her research interests are in the areas of Machine Learning, Pattern Recognition and Signal Processing and their applications to a broad range of fields.

EQREF29 local parameters INLINEFORM11 , INLINEFORM12 and INLINEFORM13 converge Compute step-size INLINEFORM14 Update topics variational parameters DISPLAYFORM0   Update annotators confusion parameters DISPLAYFORM0   global convergence criterion is met As we did for the classification model from Section SECREF4 , we can envision developing a stochastic variational inference for the proposed regression model. In this case, the only “global" latent variables are the per-topic distributions over words INLINEFORM0 . As for the “local" latent variables, instead of a single variable INLINEFORM1 , we now have two variables per-document: INLINEFORM2 and INLINEFORM3 . The stochastic variational inference can then be summarized as shown in Algorithm SECREF76 . For added efficiency, one can also perform stochastic updates of the annotators biases INLINEFORM4 and precisions INLINEFORM5 , by taking a step in the direction of the gradient of the noisy evidence lower bound scaled by the step-size INLINEFORM6 .

Regression model	In this section, we develop a variant of the model proposed in Section SECREF4 for regression problems. We shall start by describing the proposed model with a special focus on the how to handle multiple annotators with different biases and reliabilities when the target variables are continuous variables. Next, we present a variational inference algorithm, highlighting the differences to the classification version. Finally, we show how to optimize the model parameters.

Finally, for the fourth image, somehow surprisingly, the model is able to infer the correct true class, even though all 3 annotators labeled it as “inside city".

Interestingly, the figure also shows that there are a couple of workers that are considerably biased (e.g. workers 6 and 8) and that those biases are being correctly estimated, thus justifying the inclusion of a bias parameter in the proposed model, which contrasts with previous works  ,  .

Supervised topic models	Latent Dirichlet allocation (LDA) soon proved to be a powerful tool for modeling documents  and images  by extracting their underlying topics, where topics are probability distributions across words, and each document is characterized by a probability distribution across topics. However, the need to model the relationship between documents and labels quickly gave rise to many supervised variants of LDA. One of the first notable works was that of supervised LDA (sLDA)  . By extending LDA through the inclusion of a response variable that is linearly dependent on the mean topic-assignments of the words in a document, sLDA is able to jointly model the documents and their responses, in order to find latent topics that will best predict the response variables for future unlabeled documents. Although initially developed for general continuous response variables, sLDA was later extended to classification problems  , by modeling the relationship between topic-assignments and labels with a softmax function as in logistic regression. From a classification perspective, there are several ways in which document classes can be included in LDA. The most natural one in this setting is probably the sLDA approach, since the classes are directly dependent on the empirical topic mixture distributions.

Question: what datasets were used?
Answer: Reuters-21578 , LabelMe , 20-Newsgroups benchmark corpus