Machine Truth Serum

1 INTRODUCTION . Wisdom of the crowd harnesses the power of aggregated opinion of a diverse group rather than a few individuals . Though initially proposed for mainly aggregating human judgements , this idea has been successfully implemented in the context of machine learning . In particular , ensemble learning was proposed and studied to improve prediction performance by combining several learning models to obtain better results compared to a single one ( Dietterich , 2000 ) . The developed ensemble techniques have shown consistent benefits in real-world machine learning applications , evidenced by the Netflix Competition ( Bennett et al. , 2007 ) and Kaggle competition . Popular ensemble methods include Boosting ( e.g. , AdaBoost ( Freund & Schapire , 1997 ) ) , Bootstrap aggregating ( bagging ) , Stacking ( Bishop , 2006 ) , and Random Forest ( Ho , 1995 ) . The most popular , as well as simple , way to perform aggregation is via majority voting rule . The classical example is Random Forest , which outputs the majority answer from multiple trained decision trees . Inference methods ( Raykar et al. , 2010 ; Zhang et al. , 2014 ; Liu et al. , 2012 ; Zhou et al. , 2012 ; 2014 ) have been applied to perform smarter aggregation that aims to outperform majority-voted answers . These methods often leverage homogeneous assumption of certain hidden models over a large number of data points in order to perform joint inference . Nonetheless , all above methods rely on the assumption that the majority answer is more likely to be correct - this is also true for the more sophisticated inference models , as the inferences will mostly likely initiate based on majority-voted answers ( when the algorithm has no prior information ) . While enjoying this assumption that majority tends to be correct , this claim is questionable in settings where special knowledge is needed to infer the truth , but it is owned by few individuals when they are not widely shared ( Chen et al. , 2004 ; Simmons et al. , 2010 ; Prelec et al. , 2017 ) . Echoing to the above problem of aggregating human judgements , we face similar challenge when aggregating classifiers ’ predictions in machine learning . For example , we have a deep learning ( Goodfellow et al. , 2016 ) classification model which performs the best among multiple models when used in the ensemble method . For some data point , the classification result of this deep learning model may be the correct minority . In this situation , applying majority voting leads to wrong answers . We aim to complement the literature via studying whether we can aggregate classifiers better than majority voting even when majority opinion is wrong . We also target a method that can operate over each data point separately without assuming homogeneous assumptions across a massive dataset . The question sounds unlikely to resolve at a first look , but we are inspired by the seminal work Bayesian Truth Serum ( BTS ) ( Prelec , 2004 ; Prelec et al. , 2017 ) which approached this question in the setting of incentivizing and aggregating truthful human judgements . The core idea behind BTS is simple and elegant : the correctness of an answer does not rely on its popularity , but rather whether it is “ surprisingly ” popular or not - here an answer that has a higher posterior ( computed from reports of the crowds ) than its prior is taken as being “ surprisingly ” popular , and should be considered as the true answer . This argument has a very intuitive Bayesian reasoning : the signal that improves over its prior is more likely to be informative . Prelec et al . ( 2017 ) also argued that via eliciting a peer prediction information , which is defined as the fraction of “ how many other people would agree with you ” from each agent , he will be able to construct an informative prior to compare with the majority vote posterior aggregation . BTS operates over each single question separately , without seeing a large number of similar tasks ( in order to leverage a certain homogeneity assumption ) . In this paper , we make a connection between these two seemingly irrelevant topics , and extend the key idea in Bayesian Truth Serum to aggregating classifiers ’ predictions . The challenge is that we would not be able to elicit a belief from a classifier on “ how many other classifiers would agree with themselves ” , which renders the task of computing the prior difficult . We proposed two machine learning aided algorithms to mimic the procedure of reporting the peer prediction information , which we jointly name as Machine Truth Serum ( MTS ) . We firstly propose Heuristic Machine Truth Serum ( HMTS ) . In HMTS , we pair each baseline classifier ( an agent ) with a regressor model , which is trained to predict the peer prediction information using a processed training dataset . With the predictions from the regressors , we will be able to apply the idea of BTS to decide on whether to adopt the minority as the answer via comparing the prior ( computed using the regressor ) and the posterior for each label . Then we proposed Discriminative Machine Truth Serum ( DMTS ) . In DMTS , we directly train one classifier to predict whether adopting the minority as the answer or not . As for the training complexity of our algorithm , the training time of HMTS is linear in the number of label classes because of the training of extra regressors . DMTS will only need to train one additional classifier and both the training and the running time are almost the same as the basic majority voting algorithm . Therefore our proposed methods are very practical to implement and run . Our contributions summarize as follows : ( 1 ) We propose Heuristic Machine Truth Serum ( HMTS ) and Discriminative Machine Truth Serum ( DMTS ) to complement ensemble methods , which can detect when minority should be considered the final prediction instead of the majority . ( 2 ) Our experiments over 6 binary and 6 multiclass classification real-world datasets reveal promising results of our approach in improving over majority voting . Our proposed methods also outperform popular ensemble algorithms . ( 3 ) To pair with our experimental results , we also provide analytical evidences for the correctness of our proposed approaches . ( 4 ) Our approaches can be generically applied in ensemble methods to replace simple majority voting rules . The rest of the paper is organized as follows . Section 2 introduces some related works . Section 3 reviews preliminaries and BTS . Section 4 introduces our Machine Truth Serum approaches . Section 5 presents our experimental results . Section 6 concludes our paper . 2 RELATED WORK . Wisdom of the crowd ( Surowiecki , 2005 ) are often considered as being more accurate than a few elite individuals in applications including decision making of public policy ( Morgan , 2014 ) , answering the questions on general world knowledge ( Singh et al. , 2002 ) , and so on . Typical algorithms for extracting wisdom of the crowd are based on majority voting , and the assumption that the majority opinion is more likely to be correct ( Surowiecki , 2005 ) . There is another line of machine learning works on proposing inference methods , including Expectation Maximization method ( Raykar et al. , 2010 ; Zhang et al. , 2014 ) , Variational Inference ( Liu et al. , 2012 ; Chen et al. , 2015 ) , and Minimax Entropy Inference ( Zhou et al. , 2012 ; 2014 ) to crowdsourcing settings , aiming to uncover the true labels from the noisy labels provided by non-expert crowdsourcing workers . Most relevant to us , ( Prelec , 2004 ; Prelec et al. , 2017 ) proposed a Bayesian Truth Serum method to extract the subjective judgment of minority expert by collecting not only people ’ s judgements but also how many percentage of the population share the same opinion . In machine learning , ensemble methods combining multiple learning algorithms usually performs better than any single method ( Dietterich , 2000 ) . Ensemble methods consist of a rich family of algorithms . For instance , AdaBoost ( Freund & Schapire , 1997 ) and Random Forest ( Ho , 1995 ) are two different and commonly used ones . AdaBoost tries to optimize weighted voting outcomes , while Random Forest train and test using the majority voting rule . But these popular ensemble methods will be wrong when the minority is the correct answer . In both the setting of aggregating human judgements and classifiers ’ predictions , most works , except for ( Prelec , 2004 ) , would fail when the majority opinion is instead likely to be wrong . But BTS only works in the setting of aggregating human judgements by collecting subjective judgment data . Based on the ideas proposed by Prelec ( 2004 ) and Prelec et al . ( 2017 ) , we proposed two machine learning aided algorithms to find the correct answer when it is minority instead of majority in the setting of classifiers ’ predictions . As our proposed methods are machine learning algorithms , they can be trained and the predictions will be made automatically instead of collecting subjective judgment data as the case in ( Prelec , 2004 ) . 3 PRELIMINARY . In this paper , we consider both binary and multiclass classification problems . Nonetheless , for simplicity of demonstration , our main presentation focuses on binary classification . A multi-class extension of our method is presented in Section 4.3 . Suppose that we have a training dataset D : = { ( xi , yi ) } Ni=1 and a test dataset T : = { ( xi , yi ) } Ti=1 , where xi ∈ X ⊆ Rd is a d-dimensional vector . We have K baseline classifiers F : = { f1 , f2 , ... , fK : X → { 0 , 1 } } that map each feature vector to a binary classification outcome . Ensemble method such as boosting algorithms can combine { f1 , f2 , ... , fK } to get better prediction results than each single one . For instance , Random Forest first applies the bootstrap aggregating to train multiple different decision trees to correct overfitting problems of decision trees . After training , the majority rule will be applied to generate the prediction result . The above dependence on the majority voting rule is ubiquitous in ensemble methods . The key assumption of using the majority rule is that the majority is more likely to be correct than random guessing . Denoting as Maj ( { f1 ( x ) , f2 ( x ) , ... , fK ( x ) } ) the majority answer from the K classifiers , formally , most , if not all , methods require that P ( Maj ( { f1 ( x ) , f2 ( x ) , ... , fK ( x ) } ) 6= y ) < 0.5 . Our goal is still to construct a single aggregator A ( { f1 , f2 , ... , fK } ) that takes the classifiers ’ predictions on each data point as inputs and generates an accurate aggregated prediction . But we aim to provide instruction to cases where it is possible that P ( Maj ( { f1 ( x ) , f2 ( x ) , ... , fK ( x ) } ) 6= y ) > 0.5 . The challenge is to detect when the minority population has the true answer .

Summary: This paper proposes two machine learning adaptations of the Bayesian truth serum approach to aggregating predictions from human experts. The first method proposed involves training two regression models for each classifier in the ensemble that predicts the proportion of other classifiers that assign the same label to a novel instance. The second approach is to train a binary classifier that, based on the features associated with an instance, determines whether the most common or second most common prediction made by individual ensemble members should be the prediction made by the ensemble.