Machine Truth Serum
Authors: Tianyi Luo
Yang Liu
Venue:  (2020)
Keywords: Ensemble method
Classification
Machine Truth Serum
Minority
Machine Learning

Abstract: Wisdom of the crowd revealed a striking fact that the majority answer from a crowd is often more accurate than any individual expert. We observed the same story in machine learning - ensemble methods leverage this idea to combine multiple learning algorithms to obtain better classification performance. Among many popular examples is the celebrated Random Forest, which applies the majority voting rule in aggregating different decision trees to make the final prediction. Nonetheless, these aggregation rules would fail when the majority is more likely to be wrong. In this paper, we extend the idea proposed in Bayesian Truth Serum that "a surprisingly more popular answer is more likely the true answer" to classification problems. The challenge for us is to define or detect when an answer should be considered as being "surprising". We present two machine learning aided methods which aim to reveal the truth when it is minority instead of majority who has the true answer. Our experiments over real-world datasets show that better classification performance can be obtained compared to always trusting the majority voting. Our proposed methods also outperform popular ensemble algorithms. Our approach can be generically applied as a subroutine in ensemble methods to replace majority voting rule. 

Under review as a conference paper at ICLR 2020
MACHINE TRUTH SERUM
Anonymous authors Paper under double-blind review
ABSTRACT
Wisdom of the crowd (Surowiecki, 2005) revealed a striking fact that the majority answer from a crowd is often more accurate than any individual expert. We observed the same story in machine learning ensemble methods (Dietterich) (2000) leverage this idea to combine multiple learning algorithms to obtain better classification performance. Among many popular examples is the celebrated Random Forest (Ho 1995), which applies the majority voting rule in aggregating different decision trees to make the final prediction. Nonetheless, these aggregation rules would fail when the majority is more likely to be wrong. In this paper, we extend the idea proposed surprisingly more popular answer is more likely the true answer' to classification problems. The challenge for us is to define or detect when an answer should be considered as being "surprising". We present which aim to reveal the truth when is minority instead of majority who has the true answer. Our experiments performance can be obtained compared to always trusting the majority voting. Our proposed methods also outperform popular ensemble algorithms. Our approach can be generically applied as a subroutine in ensemble methods to replace majority voting rule.
Bayesian Truth Serum (Prelec) 2004) that "a
machine learning aided methods
real-world datasets show that better classification
INTRODUCTION
Wisdom of the crowd harnesses the power of aggregated opinion of diverse group rather than a few individuals. Though initially proposed for mainly aggregating human judgements. this idea has been successfully implemented in the context of machine learning. In particular, ensemble learning was proposed and studied to improve prediction performance by combining several learning models to obtain better results compared to single one (Dietterich) (200)) The developed ensemble techniques have shown consistent benefits real- world machine learning applications, evidenced by the Netflix Competition (Bennett et al., 2007) and Kaggle competition. Popular ensemble methods include Boosting (e.g.. AdaBoost (Freund & Schapire 1997)), Bootstrap aggregating (bagging), Stacking (Bishop 2006), and Random Forest (Ho 1995).
The most popular. as well as simple, way to perform aggregation is via majority voting rule. The classical example is Random Forest, which outputs the majority answer from multiple trained decision trees. Inference methods (Raykar et al.] 2010 Zhang et al., [2014] Liu et al.) 2012 Zhou et al.] 20012 2014) have been applied perform smarter aggregation that aims to outperform majority- answers. These methods often leverage homogeneous assumption of certain hidden models over a large number of data points in order to perform joint inference.
Nonetheless, all above methods rely on the assumption that the majority answer is more likely to be correct- this is also true for the more sophisticated inference models, the inferences will mostly likely initiate based on majority-vote answers (when the algorithm has no prior information). While enjoying this assumption that majority tends to be correct, this claim questionable in settings where special knowledge is needed to infer the truth, but it is owned by few individuals when they are not widely shared (Chen et al. 2004] Simmons et al.) 2010 Prelec et al., 2017). Echoing to the above problem of aggregating human judgements, we face similar challenge when aggregating classifiers' predictions in machine learning. For example, we have a deep learning (Goodfellow et al.) 2016) classification model which performs the best among multiple models when used in the ensemble method. For some data point, the classification result of this deep learning model may be the correct minority. In this situation, applying majority voting leads to wrong answers.

---

Under review as a conference paper at ICLR 2020
We aim to complement the literature via studying whether we can aggregate classifiers better than majority voting even when majority opinion is wrong. We also target a method that can operate over each data point separately without assuming homogeneous assumptions acToss massive dataset.
The question sounds unlikely to resolve at a first look, but we are inspired by the seminal work Bayesian Truth Serum (BTS) (Prelec [004 Prelec et al. 2017) which approached this question in the setting of incentivizing and aggregating truthful human judgements. The core idea behind BTS is simple and elegant: the correctness of an answer does not rely on its popularity, but rather whether it is "surprisingly" popular or not - here an answer that has a higher posterior (computed from reports of the crowds) than its prior taken as being "surprisingly" popular, and should be considered as the true answer. This argument has a very intuitive Bayesian reasoning: the signal that improves over its prior is more likely to be informative. Prelee et al. (2017) also argued that via eliciting a peer prediction information, which is defined as the fraction of "how many other people would agree with you" from each agent, he will be able to construct an informative prior to compare with the majority vote posterior aggregation. BTS operates over each single question separately, without seeing large number of similar tasks (in order to leverage a certain homogeneity assumption).
In this paper, we make connection between these two seemingly irrelevant topics, and extend the key idea in Bayesian Truth Serum to aggregating classifiers' predictions. The challenge is that we would not be able to clicit a belief from classifier "how many other classifiers would agree with themselves", which renders the task of computing the prior difficult. We proposed two machine learning aided algorithms to mimic the procedure of reporting the peer prediction information, which we jointly name as Machine Truth Serum (MTS). We firstly propose Heuristic Machine Truth Serum (HMTS). In HMTS, we pair each baseline classifier (an agent) with regressor model, which is trained to predict the peer prediction information using a processed training dataset. With the predictions from the regressors, we will be able to apply the idea of BTS to decide on whether to adopt the minority as the answer via comparing the prior (computed using the regressor) and the posterior for each label. Then we proposed Diseriminative Machine Truth Serum (DMTS). In DMTS, we directly train one classifier to predict whether adopting the minority as the answer or not. As for the training complexity of our algorithm, the training time of HMTS is linear in the number of label classes because of the training of extra regressors. DMTS will only need to train one additional classifier and both the training and the running time are almost the same as the basic majority voting algorithm. Therefore our proposed methods are very practical to implement and run.
Our contributions summarize as follows: (1) We propose Heuristic Machine Truth Serum (HMTS) and Discriminative Machine Truth Serum (DMTS) complement ensemble methods, which can detect when minority should be considered the final prediction instead of the majority. (2) Our experiments over 6 binary and 6 multiclass classification
datasets reveal promising results of our approach in improving over majority voting. Our proposed methods also outperform popular ensemble algorithms. (3) To pair with our experimental results, we also provide analytical evidences for the correctness of our proposed approaches. (4) Our approaches can be generically applied in ensemble methods to replace simple majority voting rules.
The rest of the paper is organized as follows. Section 2 introduces some related works. Section 3 reviews preliminaries and BTS. Section 4 introduces our Machine Truth Serum approaches. Section 5 presents our experimental results. Section 6 concludes our paper.
2 RELATED WORK
Wisdom of the crowd (Surowiecki) 2005) are often considered as being more accurate than a few elite individuals in applications including decision making of public policy (Morgan 2014). answering the questions on general world knowledge (Singh et al. 2002). and so on. Typical algorithms for extracting wisdom of the crowd are based on majority voting, and the assumption that the majority opinion is more likely to be correct (Surowiecki) 2005). There is another line of machine learning works proposing inference methods, including Expectation Maximization method (Raykar et al.) 2010 Zhang et al. 2014). Variational Inference (Liu et al., 2012; Chen et al., 2015), and Minimax Entropy Inference (Zhou et al.) 2012 (014) to crowdsoureing settings, aiming to uncover the true labels from the noisy labels provided by non-expert crowdsourcing workers. Most relevant to us, Prelec et al. 2017) proposed a Bayesian Truth Serum method to extract the subjective
(Prelec 2004
2

---

Under review as a conference paper at ICLR 2020
judgment of minority expert by collecting not only people's judgements but also how many percentage of the population share the same opinion.
In machine learning, ensemble methods combining multiple learning algorithms usually performs better than any single method (Dietterich, 2000). Ensemble methods consist of a rich family of algorithms. For instance, AdaBoost (Freund & Schapire 1997) and Random Forest (Ho (995) two different and commonly used ones. AdaBoost tries to optimize weighted voting outcomes, while Random Forest train and test using the majority voting rule. But these popular ensemble methods will be wrong when the minority is the correct answer.
In both the setting of aggregating human judgements and classifiers' predictions, most works, except for (Preled 2004). would fail when the majority opinion is instead likely to be wrong. But BTS only works in the setting of aggregating human judgements by collecting subjective judgment data. Based on the ideas proposed by Prelec (0044) and Prelec et al. (2017), we proposed two machine learning aided algorithms to find the correct answer when minority instead of majority in the setting of classifiers' predictions. As our proposed methods are machine learning algorithms, they can be trained and the predictions will be made automatically instead collecting subjective judgment data as the case in (Prelec 2004).
PRELIMINARY
In this paper, we consider both binary and multiclass classification problems. Nonetheless, for simplicity of demonstration, our main presentation focuses on binary classification. A multi-class extension of our method is presented in Section 4.3
Suppose that we have a training dataset == {(x., (ikk;y))) y:) and a test dataset T == ((1,, ye) yi) where 24 XX C Fd a d-dimensional vector. We have K bascline classifiers F= {fi, f2.... fK X + {0, 1}} that map each feature vector to a binary classification outcome. Ensemble method such as boosting algorithms can combine f2, f2,... fK } to get better prediction results than each single one. For instance, Random Forest first applies the bootstrap aggregating to train multiple different decision trees to correct overfitting problems of decision trees. After training, the majority rule will be applied to generate the prediction result.
The above dependence on the majority voting rule is ubiquitous in ensemble methods. The key assumption of using the majority rule is that the majority is more likely to be correct than random guessing. Denoting as Maj((fi] (f) f2(x), [[.]), fx }) the majority answer from the K classifiers, formally, most, if not all, methods require that (MMaj((f11e goal is still to construct a single aggregator A([fi, i f2- ..., fxH) that takes the classifiers' predictions on each data point as inputs and generates accurate aggregated prediction. But we aim provide instruction to cases where it is possible that P(Maj( (i (I)), f2(x)...., (x)),..,,((( + y) > 0.5. The challenge to detect when the minority population has the true answer.
f2(x), f& (x))) * y) 0.5. Our
3.1 BAYESIAN TRUTH SERUM
PPreeec 2004) considers the following human judgement elicitation problem: There are a set of agents denoted by {a;) ) The designer aims to collect subjective judgement from each agent about an unknown event E {0, 1} and aggregate accordingly. Each of the agent needs to report his {0, 1} for y, and the percentage of other agents he believes will agree with him P. E [0, 1]. We will also call this second belief information as the peer prediction information.
own predicted label
Σ Eizi 1(1=44)
Denote the belief of agent as B, P; defined as follows: p; = EEg,
K-1
We, as the designer, obtain the prediction labels (4)/K, and the percentage information {p: JKK from all the agents. The posterior for each label is defined as the actual percentage of this label which can be easily calculated utilizing the prediction results: (for label 1)
Posterior(1) = Σ, 1(1, =
(1)
K
3

---

Under review as a conference paper at ICLR 2020
Prelec al. 2017p, Prelec et al. promote the idea of using the average predicted percentage the responding label as the approximation of the priors: (for label 1).
In (Prelec 2004
K 1(4=1) (1 pt) (11=)11(1=1))) Prior(1) -i=
K
If Posterior(1) > Prior(1), label I will be taken as the surprisingly more popular answer, which should be considered as the true answer y. even though might be in minority's hands. The same rule is applied to label O. Formally, if we denote i as the aggregated answer:
1 if Prior(1) Posterior if (rior(1) Posterior(1) (1).
(3)
The rest of the paper will focus on generalizing the above idea to aggregate classifiers' predictions.
4 MACHINE TRUTH SERUM
In this section, we introduce Machine Truth Serum (MTS). Suppose we have access classifiers. Each classifier can be treated as an agent. We'd like to build a BTS-ish aggregation method to aggregate the classifiers' predictions. The challenge to compute the priors from the classifiers machine-trained classifiers do not encode beliefs as human agents do, so we cannot elicit the peer prediction information from them directly. We propose two machine learning aided approaches to perform the generation of this peer prediction information. We firstly introduce two MTS approaches for binary classification and then extend these approaches to multiclass classification case.
of baseline
4.1 HEURISTIC MACHINE TRUTH SERUM
We first introduce heuristic machine truth serum (HMTS). The high level idea is train regression model for each classifier to predict the percent of the agreement from other classifiers on the prediction of each particular data point. After getting the predicted labels and the predicted peer prediction information of the classifiers, we can again approximate the priors using the predicted peer prediction information for each classifier, compute the average and compare it to posterior. In this part, HMTS for binary classification will be introduced firstly and its multiclass extension is stated in Section 4.3.
Given the training data D {I ,,), and multiple classifiers {fy} we first try to compute the j-th classifier's s "belief" of the fraction of other classifiers that would "agree" with it. Denote this number as by for each training sample (z;,ys). De can be computed as follows:
Ej 1z (z, = fx()))
(4)
K-1
By above, we have pre- pre-processed the training data to obtain D == (( F.) 1,.., K, which can serve as the training data predict the peer prediction information of classifier (again to recall, peer prediction information is the fraction of other classifiers that classifier believes would agree with it). We then train peer prediction regression models {0 on D == {(x,, 8/ ) j = 1.... K respectively to map x; to oH. We consider different class labels and will first train two regression models: Ij.0 and 9j.1 are two belief regression models of classifier j and trained on the examples whose predicted labels are Os (DH { (/) (xx)) o) and Is (D {(x,, v/): f5 1 respectively.
Then compute the following prior of label 1 for each xy;
(x,) = { gj. (2:e)
fx;:) if (x,,) 0.
(5)
1 (x,)
After obtaining these peer prediction regression models gjs, the prior and posterior of (s) E T in the test dataset are then calculated by,
Prior(æ;, I = 1] (z(s)
(6)
K
E, (z) 1)
Posterior(z;
(7)
K
4

---

Under review as a conference paper at ICLR 2020
Algorithm I Heuristic Machine Truth Serum (Binary classification)
Require:
I: Input:
2: D {(11, y1 (Zt }: training data
3: T =[(11, 11), ...., (2r, )r) testing data
4: [Si,..., JK]: classifiers
Procedure:
5: Train K classifiers (F) on the training data
6: for j 1 to K do
7: for to do
8: Compute U/ according Eqn.44)
9: end for
Train machine belief 9j.0, 9j.1 on training dataset DJ {2 77))
11: end for
f T do
13: Compute Prior(z;, | = 1) and Posterior(z,,14 / according to Eqn. and Eqn.
if Prior(>;, 1) < Posterior(z;,, I 1) then
15:
Output "surprising" answer as the final prediction. else if Prior( 1) > Posterior(z,.
1) then Output "surprising" answer 0 as the final prediction.
17:
18:
end if
19:
end for
If Prior(z,, 1= 1) < Posterior(z;; 1), the "surprsing" answer will considered as the true answer. The decision rule similar for label 0. The procedure is illustrated in Algorithm
4.2 DISCRIMINATIVE MACHINE TRUTH SERUM
The Heuristic Machine Truth Serum above relies on training models to predict the peer prediction information for each classifier (which will be used to compute the priors) and compare them the posteriors, and then decide on whether to follow the minority opinion or not. We notice the above task of determining whether to follow the minority or not is also a binary classification question. We can therefore utilize a classification model to directly predict for each data point whether the minority should be chosen as the answer or not.
We propose Discriminative Machine Truth Serum (DMTS). Again, DMTS for binary classification will be introduced firstly and its multiclass extension is stated in Section 4.3. With DMTS, new training dataset DD = (I,, D1 J.N about whether considering the minority as the final answer or not constructed. Each data DD == (Iz, (i). for = L,...,N, N, in this new training dataset is calculated as follows: for each (z,, V1) e D
majority J on x, is different from the true label; if majority of F a;; is same the true label.
0
Now with above preparation, predicting whether majority correct or not becomes a standard classi- fication problem on DD == {24, @. This is readily solvable by applying standard techniques. In our experiments, we will mainly use a Multi-Layer Perceptron (MLP) (Goodfellow et al. 2016) denoted as F. f is trained on this new training dataset and can directly predict whether we should adopt the minority as the answer not. does not restrict to MLP and can be other classifiers. We have tried several other methods, such as logistic regression, and similar conclusions are obtained. The procedure is further illustrated in Algorithm 2 in Appendix A.3
4.3 MULTICLASS EXTENSION OF HMTS AND DMTS
HMTS and DMTS can be extended to multiclass classification problem with the same ideas by modifying them accordingly. In the multiclass case, / = {0, 10..,..,,) L} is denoted as the class need to consider different class
label of the dataset. Consider HMTS first. For each classifier labels of regression models ). where E = {0, 10,1, L}. 9j.t is the belief regression model of classifier and trained on the examples whose predicting labels are ls.
5

---

Under review as a conference paper at ICLR 2020
Again compute the following prior for each 22
g3,(z,) { (2)
if (x;) 1; if fx.) #1.
(i.f)(x;) ration
where ratio
Ie 9hela;)) is defined as the ratio of the I's belief to the summation of all the other classes' beliefs except for the predicted class utilizing majority rule.
In HMTS, Eqn. and can be modified the following:
Prior( = cc) 2j=1 97,0(12)
K
(10)
K
Posterior(z,, c) دラ K
(ff());))))
(11)
K
We then compute the priors and posteriors of each class label based on Eqn. 10 and (TI. is possible that there exist more than one class labels whose posterior is larger than its prior. We define the set containing all these label classes
Csat = {c Prior( I c) < Posterior(a;, c), E C)-
We predict the class label which has the biggest improvement from its prior to posterior:
argmax cEC Posterior(z; I c) Prior c)
In DMTS, firstly need to train model that decides whether to apply the minority the final answer which are very similar to the binary case. The difference is that we will then choose the minority answer as the predicted answer instead of using majority if i) it has the most votes in the minority answers and ii) the prediction result of classifier obtained in the training phase is (we should use minority).
4.4 THEORETICAL ANALYSIS
We conduct a formal analysis about the correctness of our proposed algorithms. Not surprisingly, the key ideas of the proofs are adapted from the proof for BTS (Prelec et al. 2017). For simplicity, we only present the theorems for binary classification. The proofs of multic lass ones are similar to the binary case. The details of proofs are left the Appendix A.
To set up for presenting the theorems, we restate our problem: we assume that each x; can take on any value the discrete set (s.,..., 8m) for the simplicity of proof. In practice, conceptually each feature vector can be represented by assigned (large-enough) categorical number. One can consider &n (A: 1, ,,, m) as a code for each feature vector. The proof based on continuous value can be deduced similarly.
Here we have two worlds Who (o = or 1) of different class labels for any zę. One world actual, the other one is counterfactual. If say wi1 is the actual world for w;, it means the predicting answer of I; in this world is and = is also the ground truth label of x;. Wao is the counterfactual world and the predicting answer of x; in this world is O. In this paper, While finite samples is practical setting, is important to first analyze and conclude some deductions in the infinite sample ideal case.
considering infinite samples.
Theorem 4.1. No algorithm exists for deducting the correct classification answer relying exclusively on feature vector distribution of true class label, P(sk Bo 3o-)), 1..., m and correctly computed posterior distribution over all possible classification labels given feature vectors, P(yo)sk), 1.... m. 0, for any I,, o" the true class label.
Theorem 4.2. For any Ii, the average estimate of the prior prediction for the correct classification answer will be underestimated if not every classifier provides the correct classification prediction. Therefore, the minority should be the final answer instead of the majority if the prior (estimated prediction) less than the posterion.
Theorem indicates that exclusively posterior probabilities based methods such as majority voting can not infer the true answer for all the time. Theorem 2, the posterior and prior are the prediction distribution of other classifiers for each classifier both are provided by proposed MTS algorithms.

---

Under review as a conference paper at ICLR 2020
Complexity For HMTS, for example in our experiments, another 15 (L + 1) (label classes {0, I....,L,,, L)) simple regressors will be trained to predict others' beliefs based on 15 baseline classifiers. So the total training time is linear in the number of label classes. After training the extra regressors, running the algorithm only requires taking +++ averages (15 of the 15 (I. + 1) regressors each) and compare with average posterior. DMTS will only need train additional classifier based on 15 classifiers and both the training and the running time are almost the same as the basic majority voting algorithm. The above complexity analysis shows our methods are very practical.
EXPERIMENTS
In this section, we present our experimental results. Particularly we test our proposed two MTS algorithms on 6 binary and 6 multiclass real-world classification datasets. Experimental results show that consistently better classification accuracy can be obtained compared to always trusting the majority voting outcomes.
5.1 DATASETS
In this section, 6 binary and 6 multiclass classification benchmark datasets (Pang & Lee 2004) are used to conduct the experiments. The statistical information of these datasets are described in Table in Appendix [A.2] In this paper, each of the datasets we used has small size we chose to focus on the small data regime where the classifiers are likely to make mistakes. This is a better fit to our setting where majority opinion can be wrong with good chance. For the splitting of training and testing, we used the original setting for the datasets providing training and testing files separately. For other datasets, only data file is given. For the testing results' statistical significance, more data is distributed to testing dataset and 50/50 is considered as the splitting of training and testing.
5.2 EXPERIMENTAL SETUP AND RESULTS
Table Overall accuracy and the number of increased correct predictions in the "high disagreement" cases of toifformly-eeighee Majority Voting, HMTS, and DMTS on the 6 binary classification datasets. The "high disagreement" means that the difference between the number of predicting 0 and I is small. We have 15 classifiers and the instance will be considered having "high disagreement" if the vote number of majority class is or 9. In HMTS, the numbers of "high disagreement" instances consider in 6 datasets 51, 52, 37, 149, 43, 45. As for DMTS, the numbers 30, 64, 37, 76, 13, 38. The results having the highest accuracy for each dataset are highlighted.
Datasets
Breast cancer
Hill Valley
Movic Review
Spambase 73.57%
Australian
German
Majority
92.96%
80.86%
80.10%
81.74%
76.00%
HMTS
96.13% (+8/51) 81.69% (+3/52)
80.85% (+5/37)
76.87% (+48/149)
83.44%
(+6/43)
77.20% (+6/45)
DMTS
94.01% (+4/30)
81.03% (+1/64) 80.60% (45/37) 77.35% (+70/76)
82,94% (+4/13) 76.20% (+1/38)
In our binary classification experiments. consider commonly used binary classification al- gorithms which are Perceptron (Rosenblati 1958), Logistic Regression (LR) (Peng et al. 2002). Random Forest (RF), Support Vector Machine (SVM) (Chang & Lin 201T), and MLP. In order to test the usefulness of our methods, we experiment with a noisy environment we flipped the true class label with three noisy rates construct three binary classifiers for each of the 5 methods which have mediocre performance on the test datasets. We wanted to diversify our classifiers by introducing different noisy rates (varying the data distribution). Our experiments used 0.06, 0.08, and 0.1 (probability of flipping the label) for each family of classifier. We also tried other values such as 0.1, 0.2, and 0.3, and we reached similar conclusions. In total, 15 different classifiers are obtained as the baseline classifiers.
The experimental results on the 6 binary classification datasets are reported in Table | From these results, observe that Heuristics Machine Truth Serum (HMTS) tends to have more robust and better performances than Discriminative Machine Truth Serum (DMTS) in most datasets, especially in the small-size datasets. These can be explained by the fact DMTS itself is ML. classifier which
7

---

Under review as a conference paper at ICLR 2020
needs larger size of data get good results. That HMTS improve the classification accuracy in the small size of dataset is particularly useful in some fields such as healthcare in which collecting data is very tmm-cconuming and expensive. As for the running time, DMTS is faster than HMTS as HMTS needs to compute the peer prediction results all the 15 classifiers and DMTS only predicts once.
Table 2: Overall accuracy and the number of increased correct predictions the "high disagreement" cases of Uniformly-weighteed Majority voting, HMTS, and DMTS the 6 multi-class classification datasets. We have 15 classifiers and the instance will considered as having "high disagreement" if the vote number of majority class is less equals to 6 for the 3-class and 4-class datasets. The threshold number is 5 for 6-class and 3 for 10-class datasets. In HMTS, the numbers of "high disagreement" instances consider in datasets 61, 54, 27, 65, 15, 157. As for DMTS, the numbers are 48, 45, 25, 81, 25, 40. The results having the highest accuracy for each dataset are highlighted.
Datasets of class
Abalone 3
Waveform
Wall-Following
Statlog
Optical
Pen-Based
10
10
Majority
51.25%
85.04%
90.22%
86.70%
97.50%
95.08%
HMTS
51.45% (44/61)
85.48%
(+11/54)
90.26% (+1/27)
87.10%
(48/65)
97.61% (42/15)
95.57% % (+17/157)
DMTS
51.40% (43/48) 85.60% (+14745) 90.59% (+10/25) 86.75% (+1/81) 97.66% (+3/25)
95.51% (+15740)
We also tested our extension to multi-class classification problems. Experimental results multi- class classification datasets are reported in Table 2 We observe that HMTS and DMTS obtained similarly good performance in the accuracy metric because the size of multi-class classification datasets is larger and the MLP of DMTS can perform better than the binary case.
Table 3: Comparison between popular ensemble and our proposed approaches
Random Forest
Weighted Majority
Methods
Adaboost
Stacking
HMTS
DMTS
Breast Cancer
94.37%
94.37%
94.01%
94.72%
96.13%
94.01 %
Hill Valley
62.05%
57.93%
53.30%
62.38%
81.69 %
8 81.03%
Movie Review
75.10%
77.20%
81.60%
70.30%
80.85%
80.60%
Spambase 74.74%
74.65%
74.17%
75.91% %
76.87%
77.35
Australian 82.03%
84.06%
84.06%
85.22%
83.44%
82.94%
German 72.20%
74.80%
73.80%
77.20%
77.20%
76.20%
Abalone 48.70%
54.25%
55.35%
54.55%
51.45%
51.40%
Waveform 81.80%
82.60%
85.36%
84.00%
85.48%
85.60 %
Wall-Following 99.00%
95.22%
95.37%
89.07%
90.26%
90.59%
Statlog 85.85%
86.15%
86.85%
82.70%
87.10%
86.75%
Optical 93.99%
94.88%
92.21%
95.83%
97.61%
97.66
Pen-Based 94.97%
95.45%
90.59%
95.43%
95.57% %
95.51%
Finally, we compare between several popular ensemble algorithms and our proposed approaches. We list the testing accuracy for Adaboost with 15 decision tree base estimators, Random Forest with 15 decision trees, Weighted Lajority(Germall et al. 2015). Stacking with the same setting of 15 classifiers utilized in our two MTS algorithms and Logistic Regression or SVM as meta classifier, HMTS, and DMTS for all datasets Table 3 shown in the table, HMTS and DMTS outperform Adaboost, Random Forest, Weighted Majority, and Stacking in 8 datasets and are very close to the best in 3 datasets. An outlier dataset is Wall-Following and we found that decision tree based methods can get perfect performance on it. Compared to other weighted methods, we'd like to note that our aggregation operates on each single task separately this means that our method will be more robust when the difficulty levels of tasks differ drastically in the dataset. None of the other weighted methods (with fixed and learned weights) has this feature. We also find that our method is robust smaller number of classifiers, in contrast to, say Adaboosting.
8

---

Under review as a conference paper at ICLR 2020
6 DISCUSSION AND CONCLUDING REMARKS
This paper proposes two machine learning aided methods HMTS and DMTS to detect when the minority should be the final answer instead of majority. Our experiments multiclass real-world datasets show that better classification performance can be obtained compared always trusting the majority voting. Our proposed methods also outperform popular ensemble algorithms on three randomly selected datasets and can be generically applied as subroutine in ensemble methods replace majority voting. For future work, we plan try more types of classifiers, especially the recent deep learning models, to train the belief models for baseline classifiers and apply our methods more real-world datasets.
6 binary and 6
REFERENCES
James Bennett, Stan Lanning, al. The netflix prize. In Proceedings of KDD cup and workshop. volume 2007, pp. 35. New York, NY, USA., 2007.
Christopher M Bishop. Pattern recognition and machine learning. springer, 2006.
Chih-Chung Chang and Chih-Jen Lin. Libsvm: A library for support vector machines. ACM transactions on intelligent systems and technology (TIST), 2(3):27, 2011.
Kay-Yut Chen, Leslie Fine, and Bernardo A Huberman. Eliminating public knowledge biases in information-aagregaull mechanisms. Management Science, 50(7):983-994, 2004.
Xi Chen, Qihang Lin, and Dengyong Zhou. Statistical decision making for optimal budget allocation in crowd labeling. The Journal of Machine Learning Research, 16(1):1-46, 2015.
Thomas Dietterich. Ensemble methods in machine learning. In International workshop multiple classifier systems, pp. 1-15. Springer, 2000.
Yoav Freund and Robert E Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of computer and system sciences, 55(1):119-139) 1997.
Pascal Germain, Alexandre Lacasse, Francois Laviolette, Mario Marchand, and Cean-Francis Roy. Risk bounds for the majority vote: From pac-bayesian analysis to a learning algorithm. The Journal of Machine Learning Research, 16(1):787-860, 2015.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Tin Kam Ho. Random decision forests. In Proceedings of 3rd international conference on document analysis and recognition, volume pp. 278-282. IEEE, 1995.
Qiang Liu, Jian Peng, and Alexander T Ihler. Variational inference for crowdsourcing. In Advances in neural information processing systems, pp. 692-700, 2012.
M Granger Morgan. Use (and abuse) of expert elicitation in support of decision making for public policy. Proceedings the National Academy of Sciences, 111(20):176-7188) 2014.
Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summa- rization based on minimum cuts. In Proceedings of the 42nd annual meeting on Association for Computational Linguistics, pp. 271. Association for Computational Linguisties, 2004.
Chao-Ying Ying Joanne Peng, Kuk Lida Lee, and Gary M Ingersoll. An introduction logistic regression analysis and reporting. The journal of educational research, 96(1):3-14, 2002.
Dražen Prelec. A bayesian truth serum for subjective data. science, 306(5695):462 466, 2004.
Dražen Prelee, H Sebastian Seung, and John McCoy. A solution to the single-question crowd wisdom
problem. Nature, 54167688):532, 2017.
Vikas C Raykar, Shipeng Yu. Linda H Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. Learning from crowds. Journal of Machine Learning Research, 11(Apr): 1297-1322, 2010.

---

Under review as a conference paper at ICLR 2020
Frank Rosenblatt. The perceptron: probabilistic model for information storage and organization in the brain. Psychological review, 65(6):386, 1958.
Joseph P Simmons, Leif D Nelson, Jeff Galak, and Shane Frederick. Intuitive biases in choice versus estimation: Implications for the wisdom of crowds. Journal of Consumer Research, 38(1):1-15, 2010.
Push Singh, Thomas Lin, Erik T Mueller, Grace Lim, Travell Perkins, and Wan Li Zhu. Open mind common sense: Knowledge acquisition from the general public. In OTM Confederated International Conferences" On the Move to Meaningful Internet Systems", pp. 1223-1237. Springer, 2002.
James Surowiecki. The wisdom of crowds. Anchor, 2005.
Yuchen Zhang, Xi Chen, Dengyong Zhou, and Michael I Jordan. Spectral methods meet em: A provably optimal algorithm for crowdsourcing. In Advances in neural information processing systems, pp. 1260-1268, 2014.
Dengyong Zhou, Sumit Basu, Yi Mao, and John C Platt. Learning from the wisdom of crowds by minimax entropy. In Advances in neural information processing systems, pp. 2195-2203, 2012.
Dengyong Zhou, Qiang Liu, John Platt, and Christopher Meek. Aggregating ordinal labels from crowds by minimax conditional entropy. In International conference on machine learning, pp. 262-270, 2014.
A APPENDIX
A.1 PROOF OF THROREMS IN SECTION 4.4
In this part, provided the detailed proof of two theorems which are the analytical evidences for the correctness of our proposed approaches, For simplicity, we only show the proof details of binary classification. The proof of multiclass classification is similar to the binary case. This proof is largely adapted from (Prelec et al. 2017). Nonetheless we reproduce the details for completeness.
Theorem A.1.
No algorithm exists for inferring the correct classification answer relying exclusively on feature vector distribution of true class label, P(sk you ) = I.., m and correctly computed posterior distribution over all possible classification labels given feature vectors, P(BolSk), ,,, m, 0 0, 1. for any Ii. o" the true class label.
Proof. In this proof, for any arbitrarily selected class label, we can construct a world model in which this selected class label is predicted as the answer and it also the ground truth class label. And this world model can also generate feature vector distribution of true class label and correctly computed posterior distribution over all possible classification labels given feature vectors.
Based on the description of theorem, (( 3y ), k 1,.., and P ([yo|sk), are known. But we don't 't know which class label is the correct answer you We can arbitrarily selected any class label yo the ground truth class label. In the following part, corresponding world model (skk. yo) which can generate the known P(sk yo-) and P(yol8k) will be constructed.
.... m, 0, 1
Because the known parts don't 't constrain the prior the feature vector these priors can model differences in the baseline classifiers. In particular, we can set the prior to:
((sx)= P(sklyo-) (k)))s))
P(srlyo-) k 1,.., P(y.ls,)
Σ
Because posteriors in the constructed world model must equal known posteriors: Qvvolsk) (()o(s)), for k = 1,..., o = 0, 1. So we can get the joint distribution of answer and the feature vector $ę in the constructed world model:
P(sp|yo) Wor P(yo|18)
Q(uo, sk) = Quvo (3).)1)))00 ((sk)) = P(5a (ool
Σ
10

---

Under review as a conference paper at ICLR 2020
Then get the marginal distribution Yo in the constructed world by summing over k:
Q(y))= Σ P(suluo-) (I P(
P(srlBo-) P(yo(sp)
P(y.la.)
&
After getting the marginal distributions (5x), Q(yo), and the matching posteriors, Qvoolsa) (8), for I.... m. the feature vector distribution of true class label in the constructed world, Q (Sk Bo) be calculated by:
((o) Q(volss)Q(sm) P(sulyo-) Q(yo)
Because Yo arbitrarily chosen, this theorem is proved.
Theorem shows that any algorithm relying exclusively on feature vector distribution of true class label and correctly computed posterior distribution over all possible classification labels given feature vectors (e.g. majority voting) can not deduct the correct classification answer.
In the following part, we are considering the extra information which is the estimation of other classifiers' prediction results. We use (s)) to represent the how many percentage of classifiers will predict Yo given Sg. We also define world classification function W (s&) ((w,.ss). Two thresholds and = 1 - are given to make the final classification result. The classification rule as follows:
- wo if P(wal(s) (8k) > cu: w1 if P(w1 (sk) > c1.
((x))
Theorem A.2. For any Is the average estimate of the prior prediction for the correct classification answer will be underestimated if not every classifier provides the correct classification prediction.
Proof. We first prove that the actual percentage of correctly predicted classifiers for the true answer in the actual world exceeds counterfactual world's percentage for the true answer, P(u-- (wor) > P(u wx), k + o".
By the definition of W(sk), we can get P(wo- (Vo-) > Cor, P(wo- (ux) Cor. Then we have P(wo- (0o-) P(vx) > P(wo- (4 (xx)) P(rx). So
P(w (o-)) > Pw |vo P((o))) + P(wo- (ux) P(wo))
(12)
According to Bayesian rule, have the following deduction:
P(u ( P(w-- |vo- P(Ww)
P(w (6o.) 1 1 --P(w-))
(13)
P(u [ P(wy vo- P(ww))
T- P(wo-|Wo-) P(wo-)
Based on 12
13 is greater than one. P(us (wo.) > P(vo- P(vo--lke) h + is proved.
The estimate of classification prediction given the feature value sj can be computed by marginalizing the actual and counterfactual worlds, P(v.- (sy) P(u.- | P ((
(s;) P(v-- (w P(wx (sy).
And we proved that P(u,- (W,)) > P(u,- (2)) k o o". Therefore, P(o--1s) Isj) < P( (w,-) It will be the strict inequality unless P(w,- 18,) = Because some feature vectors will lead to strict inequality, the average estimate of the prior prediction will be strictly underestimated. This theorem is proved.
A.2 DATASETS
In this section, the detailed information 6 binary and 6 multiclass classification datasets are described in Table ##' of Inst. stands for the number of instances. '#" of Attr. stands for the number of attributes. "% of Maj stands for the percentage of majority class.
A.3 ALGORITHM 2 IN SECTION 4.2
In this part, we provided the detailed algorithm description for DMTS in Algorithm
11

---

Under review as a conference paper at ICLR 2020
Table 4:
Statistics of 6 binary and multiclass classification datasets
Data set
Breast cancer
Movie Review
German
Australian
Hill Valley
Spambase
# of Inst.
569
1000
1000
690
606
4601
# of Attr.
30
77
24
14
100
57
% of Maj.
62.7%
50%
70.0%
67.8%
50.7%
60.6%
Data set
Abalone
Waveform
Wall-Following
Stalog landsat
Optimal
Pen-Based
# of Inst.
4177
5000
5456
4435
3823
7494
# of Attr.
8
21
24
36
62
16
% of Maj.
34.6%
33.9%
40.4%
24.2%
10.2%
10.4%
Algorithm 2 Discriminative Machine Truth Serum (Binary classification)
Require:
Input:
2: D {(11,1); (INN, ):)): training data
3: T {(211 11), (IT, yr) testing data
Procedure:
4: for to do
5: Compute Y; according to Eqn.
6: end for
7: Train DMTS classifier on the dataset { Di J.V. i==
8: for /= = to T do
Compute the classification result D. F(xe) 10: if' J, = 0 then
9:
Stay with the majority answer.
12:
else if y, then
13:
Predict with the minority answer.
14:
end if
15:
end for
12


Reviews:
[{"review_id": "rkgMptKk5B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update after author response:\nI would like to thank the authors for the thoughtful response, and for addressing several of the concerns raised by the reviewers. The updated draft look cleaner and conveys the value of the proposal better. I am changing my assessment to \"6: Weak Accept\" (the smallest jump from 3 is to 6 in the portal. I would choose a 5 if that was possible). My concerns for the significance of results and lack of a baseline that takes worker quality into account still stand, but the results look better and the authors make a more convincing case prompting me to change my rating.\n---------------------------\n\nIn this paper, the authors propose a way to measure a notion of surprise (disparity between prior and posterior) and using that as a classification rule. So, if the posterior for a class is larger than its prior, the method outputs that class label. In order to estimate the average prior for a pool of models, the method recommends building additional models to predict these “peer-priors”, whereas the posterior is simply estimated by a maximum likelihood model (P(y=1) = Indicator(y=1)/K). The authors also propose a variant that directly tries to predict when the majority answer might be wrong. The authors show that such methods can do better than majority voting and some ensemble methods on a few datasets.\n\nThe paper looks at an interesting problem of when a majority vote response might be wrong by extending the notion of surprise previously defined for human labelers to machine learning models. The strong points of the paper:\n1. Simple and interpretable extension of a previously studied method. \n2. The method can be plugged into existing frameworks to replace majority voting.\n3. The results on the datasets considered seem good.\n\nHere are some of my concerns:\n1. The baseline of majority voting is fairly weak, since there are several models that take worker quality into account when aggregating responses. For example, the classic Dawid-Skene (1979) model. It’s not very promising to see a model just beat majority voting.\n2. The presentation of results in Table 1 and 2 can be improved. Even though it’s good to know how many predictions were corrected, it will also be good to see the overall accuracy numbers (like the ones Table 3).\n3. DMTS doesn’t really have the same underlying machinery as HMTS, since it doesn’t operate on the surprise measure, and putting them together in the same paper dilutes the focus. \n4. What are the weights in the weighted majority?\n5. The results in Table 3 show that HMTS is better than other ensemble methods but only marginally. Since HMTS uses more complex intermediate models (MLPs), I am not convinced whether the small improvement is from the proposed method or just more expressive models. For example, what would happen if the base classifiers in Adaboost were MLPs?\n6. How is minority defined in multi-class scenario?\n7. Minor formatting and grammatical issues: “likely to the correct”, “majority is tending to be correct”, “seemingly irrelevant topics”, “whether adopting the minority”, “aim to provide instruction to cases”, “linear regression” (should be “logistic regression” since it’s a classifier), and so on.\n\nIn summary, I think the paper has an interesting approach to an important problem, but with results that are only marginally convincing. I would have liked to see a more thorough empirical investigation to clearly establish the value of the proposed method. Based on these observations, I think the paper misses the mark and is slightly below the acceptance threshold for me.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BklV_RvRtB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Inspired by work in ensembling human decisions, the authors propose an ensembling technique called \"Machine Truth Serum\" (based off \"Bayesian Truth Serum\"). Instead of using majority vote to ensemble the decisions of several classifiers, this paper follows the \"surprisingly popular\" algorithm; the ensembled decision is the decision whose posterior probability (based on several classifiers) most exceeds a prior probability (given by classifier(s) trained to predict the posterior predictions). It's quite a nice idea to bring this finding from human decision-making to machine learning. If it worked in machine learning, it would be quite surprising, as the surprisingly popular algorithm risks that the ensemble makes a decision against the majority vote, which is usually consider the safe/default option for ensembling.\n\nOverall, I did not find the experiments (in the current state) to provide compelling enough support for the claim that MTS is a useful approach to ensembling in machine learning. \n* Unless I am mistaken, the authors use a more powerful model (an MLP) as the regressor compared to some of the models they ensemble over. In practice, people ensemble the most powerful models they have available, so it's unclear if using a regressor with the same capacity as the ensembled classifiers will provide any additional benefit. On a related note, it would be nice to know what is the classification performance of each individual classifier? As well as how often the regressor correctly choose to go with several weaker models rather than the strongest model. In particular, I am concerned that the performance of the ensemble might be less than or equal to the performance of a single MLP classifier (or whatever other model does best).\n* \"In this paper, each of the datasets we used has a small size - we chose to focus on the small data regime where the classifiers are likely to make mistakes.\" Why not try large data tasks that are challenging for state-of-the-art models? The paper makes a general claim that MTS is a good way to aggregate predictions, so only evaluating on small datasets seems to be a limitation\n* As I understand (correct me if I am wrong), the reported results are only on examples with \"high disagreement\" between classifiers. However, for practical use cases, it is useful to know how the overall accuracy compares. One major risk of using the \"surprisingly popular\" algorithm is that the algorithm may cause the ensemble to make many incorrect predictions when the majority is right (but the minority prediction is selected). If you have those numbers, I would be interested to see them added to the paper.\n\nI am also unsure about if applying the \"surprisingly popular\" algorithm in machine learning makes sense. The algorithm is motivated by the fact that difference agents have different information. However, in the ML setting, various classifiers usually have the same information. It's possible to restrict the information given to each classifier, but that would limit the performance of each individual classifier (and hurt the ensemble). I would be curious if the authors have any thoughts on this point.\n\nI also have a few questions/concerns about how the approach is implemented:\n* Why not train a single model to predict the average prediction of all models and use that model's prediction as the prior? This approach seems simpler but equivalent to the approach currently taken.\n* Why not use model distillation (predicting all output logits/probabilities, or an average thereof) rather than just predicting an average of 0/1 predictions?\n* For HMTS, why do the regressors for each of L labels need to be separate? It seems more efficient to use a multi-class model (as many model distillation approaches do)\n* If DMTS can learn to predict when most classifiers are wrong, why wouldn't the original classifiers themselves learn to predict the answer correctly? It seems to me that the reason the experiments show that DMTS/HMTS work is that some/many of the underlying classifiers are weaker than the model that is used to ensemble the predictions (an assumption that doesn't hold in practice).\n\n\nOverall, I really like the high-level idea, and a better ensembling approach promises to bring empirical gains across many ML tasks. However, I have several concerns about the experiments, motivation, and algorithmic decisions which make me hesitant to recommend the paper for acceptance.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJlDWqa2Fr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes two machine learning adaptations of the Bayesian truth serum approach to aggregating predictions from human experts. The first method proposed involves training two regression models for each classifier in the ensemble that predicts the proportion of other classifiers that assign the same label to a novel instance. The second approach is to train a binary classifier that, based on the features associated with an instance, determines whether the most common or second most common prediction made by individual ensemble members should be the prediction made by the ensemble.\n\nPros:\n+ The core idea of adapting Bayesian truth serum to ensemble prediction in machine learning seems sensible\n+ There is some evidence that the methods have an advantage over other common ensemble approaches in practice\n+ Although there are quite a few small English mistakes, the paper is well structured and generally quite easy to follow\n\nCons & questions:  \n1. The theorem statements and proofs are underwhelming. First they seem quite vague. Second, it’s not clearly spelled out how the theorems relate to the presented method, and whether they really says anything useful about its correctness or efficacy. At face value they do not obviously analyse the correctness of the algorithm as claimed on pg6. \n2. The paper contains some simple experiments, but I do not believe they are an adequate enough evaluation of the proposed approaches. The most useful comparison are the results given in Table 3, but only three datasets are used, the margins are small, error bars are not provided, and no significance testing is performed. \n2.1 Standard practice when comparing multiple classifiers on multiple datasets would be to employ Friedman/Nemenyi post-hoc tests to determine relative performance of methods---see \"Statistical Comparisons of Classifiers over Multiple Data Sets\" by Janez Demšar (JMLR, 2006).\n2.2 At minimum we expect Tab 3 to report results for all datasets used in the earlier experiments.\n2.3 The experiments compare with AdaBoost, Random Forest, and Weighted Majority. I feel that stacking is probably the most interesting baseline to compare with, as this is a method for learning how to aggregate predictions from ensemble members.\n\nOther: \nA. This paper does not directly deal with representation learning, so is only loosely relevant to ICLR.\nB. Not clearly unpacked why regression models need to be trained to predict \\hat{y}_i^j (Equation 4) when this quantity can be computed at test time without knowing the ground truth label?\nC. I do not understand the \"X out of Y\" description given in the caption to Table 1, which makes the results in this table hard to interpret. What is meant by \"classifiers' disagreement is high enough\"?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a family of new methods, based on Bayesian Truth Serum, that are meant to build better ensembles 
from a fixed set of constituent models.

Reviewers found the problem and the general research direction interesting, but none of the three of them were convinced that the proposed methods are effective in the ways that the paper claims, even after some discussion. It seems as though this paper is dealing with a problem that doesn't generally lend itself to large improvements in results, but reviewers weren't satisfied that the small observed improvements were real, and urged the authors to explore additional settings and baselines, and to offer a full significance test.
Decision: Reject