SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Notation Description
Q,ˆQ	Q denotes the distribution over the clean states.ˆQ denotes the distribution over the noisy states.
w,ˆw	w denotes the paper's clean state.ˆw denotes the paper's noisy state.
Xi,ˆXi xi,ˆxi	Xi denotes a random variable indicating reviewer i's clean signal for a paper which state follows Q.ˆXi denotes a random variable indicating reviewer i's noisy signal for a paper which state follows Q. xi,ˆxi are their realizations respectively.
U,ˆU	U denotes the prior joint distribution over two reviewers' ratings, where Us,t = PrQ[Xi = s, Xj = t].ˆU denotes its noisy version thatˆUs,t = PrˆQ[Xi = s, Xj = t].
q,ˆq	q denotes the prior distribution vector over a reviewer's vote, where qs = PrQ[Xi = s].ˆq denotes its noisy version thatˆqs = PrˆQ[Xi = s].
P,ˆP	P denotes the prediction matrix, where Ps,t = PrQ[Xj = t|Xi = s], i ≠ j.ˆP denotes its noisy version whereˆPs,t = PrˆQ[Xj = t|Xi = s], i ≠ j.
n	n denotes the number of reviewers that are assigned for a paper.
M	M is the noise matrix defined by M = (1 - λ)I + λIb, where λ is the noise level and b is the bias vector.
v,ˆv	The frequency vector of the ratings, where vs = 1/n ∑i 1[xi = s].ˆv denotes its noisy version thatˆvs = 1/n ∑i 1[ˆxi = s].
S*(·)	S*(·) is our Surprisal-based Score calculated from w and U.
S(·)	S(·) is our Empirical Surprisal-based Score calculated from n agents' ratings and predictions.
Table 1: Notation table
Claim 3. In the binary case  $(\Sigma = \{0(\text{reject}), 1(\text{accept})\})$ , the  $\mathcal{M}^*$ -invariant surprisal vector can be simplified as
$$
\operatorname {S u r p} ^ {*} (\mathbf {w}, \mathbf {U}) = \frac {\mathbf {w} - \mathbf {q}}{\sqrt {q _ {0} q _ {1} (P _ {1 , 1} - P _ {0 , 1})}}.
$$
Theorem 1 (Invariance). The  $\mathcal{M}^*$ -invariant surprisal is non-negative, vanishes when the state  $\mathbf{w}$  equals the prior  $\mathbf{q}$ , and is invariant to  $\mathbf{M}_{\lambda,\mathbf{b}}$  for all  $\lambda \in [0,1)$ ,  $\mathbf{b} \in \Delta_{\Sigma}$ . The implication of invariance is that for any noise with  $\lambda \in [0,1)$ ,  $\mathrm{Surp}^*(\mathbf{w},\mathbf{U}) = \mathrm{Surp}^*(\hat{\mathbf{w}},\hat{\mathbf{U}})$ .
The proofs of Claim 2, Claim 3 and Theorem 1 are deferred to Appendix C. For general noise, the direction of  $\mathbf{w} - \mathbf{q}$  may not be invariant. For example, one possible noise is flipping the ratings, i.e., rating "accept" when "reject" and rating "reject" when "accept". In this case, the direction of  $\mathbf{w} - \mathbf{q}$  will be flipped as well. However, if we only care about the amount of surprisal, we successfully design a measure that is invariant to all possible non-degenerate noise and show that it can be used to identify the true state  $\mathbf{w}$  in some scenarios (see Appendix B).
4 Surprisal-based Score
This section introduces our scoring process based on the surprisal vector introduced in Section 3. We first introduce the score based on "wishful thinking", i.e., when we have the state (or the noisy state). We then resolve this "wishful thinking" by providing an estimation of the state.
Definition 3 (Surprisal-based Score). Given  $\mathbf{w}$  and  $\mathbf{U}$ , we define the  $\mathcal{M}^*$ -invariant Surprisal-based Score as
$$
\mathrm {S} ^ {*} (\mathbf {w}, \mathbf {U}) = \left(\mathbb {E} _ {\mathbf {w}} \varphi - \mathbb {E} _ {\mathbf {q}} \varphi\right) \det  (\mathbf {U}) ^ {- \frac {1}{2 (| \Sigma | - 1)}}.
$$
According to the simplification in the binary case (Claim 3), the  $\mathcal{M}^*$ -invariant surprisal score in the binary case can be simplified to
$$
\mathrm {S} ^ {*} (\mathbf {w}, \mathbf {U}) = \frac {w _ {1} - q _ {1}}{\sqrt {q _ {0} q _ {1} (P _ {1 , 1} - P _ {0 , 1})}}.
$$
Theorem 1 directly implies the following results: given the noisy states, we can compare their Surprisal-based Scores  $\mathbf{S}^{*}(\mathbf{w},\mathbf{U})$  and the comparison is invariant to the noise in  $\mathcal{M}^*$  and consistent to the comparison of their true qualities.
Corollary 1 (Noise-invariant comparison by  $\mathrm{S}^*(\cdot)$ ). For any two states  $\mathbf{w}_A, \mathbf{w}_B$  which follow distribution  $Q$ , and any two noises  $M_A, M_B \in \mathcal{M}^*$ ,
$$
\begin{array}{l} \mathrm {S} ^ {*} \left(M _ {A} (\mathbf {w} _ {A}), M _ {A} (\mathbf {U})\right) - \mathrm {S} ^ {*} \left(M _ {B} (\mathbf {w} _ {B}), M _ {B} (\mathbf {U})\right) \\ = \mathrm {S} ^ {*} (\mathbf {w} _ {A}, \mathbf {U}) - \mathrm {S} ^ {*} (\mathbf {w} _ {B}, \mathbf {U}) \\ \propto \mathbb {E} _ {\mathbf {w} _ {A}} \varphi - \mathbb {E} _ {\mathbf {w} _ {B}} \varphi . \\ \end{array}
$$
When there are only a small number of reviewers, we will use an estimation of the Surprisal-based Score, which we call the Empirical Surprisal-based Score. We first focus on the binary case. We have already proved that for any invertible noise matrix  $\mathbf{M}$  where  $M_{1,1} > M_{0,1}$ , the metric  $\frac{w_1 - q_1}{\sqrt{(q_0q_1(P_{1,1} - P_{0,1}))}}$  can be used to implement a noise-invariant comparison (Claim 2, Corollary 1). We will use the reviewers' ratings and predictions to estimate the score. We employ a frequency vector  $\hat{\mathbf{v}} \in [0,1]^{1 \times |\Sigma|}$  to denote the frequency of the reviewers' reported signals. Formally, for all  $s \in \Sigma$ ,  $\hat{v}_s = \frac{1}{n} \sum_{i=1}^n \mathbf{1}[\hat{x}_i = s]$ , where  $\hat{x}_i$  is the reported noisy signal of reviewer  $i$ . For example, in the binary case,  $\hat{\mathbf{v}} = (\hat{v}_0, \hat{v}_1)$  where  $\hat{v}_1$  is the fraction of "accept" and  $\hat{v}_0$  is the fraction of "reject".
Empirical Surprisal-based Score (Binary) When there exists at least one "accept" and one "reject", we construct a  $2 \times 2$  prediction matrix  $\hat{\mathbf{P}}$  based on the reviewers' ratings and predictions. Each element of the matrix,  $\hat{P}_{s,t}$ , is the average prediction from the reviewers who report the signal  $s$ , for the probability that a random reviewer reports the signal  $t$ . For example,  $\hat{P}_{0,1}$  is the average of the negative reviewers' prediction for the probability a random reviewer votes for "accept". When reviewers' ratings are the same, the matrix  $\mathbf{P}$  cannot be constructed. In this case, we set  $+\infty$  score for the papers that all reviewers vote for "accept" and  $-\infty$  for the papers that all reviewers vote for "reject". Formally,
$$
\mathrm {S} \big ((\hat {x} _ {i}, \hat {\mathbf {p}} _ {i}) _ {i \in [ n ]} \big) = \left\{ \begin{array}{l l} - \infty & \hat {v} _ {1} = 0 \\ \frac {\hat {v} _ {1} - \hat {q} _ {1}}{\sqrt {\hat {q} _ {0} \hat {q} _ {1} (\hat {P} _ {1 , 1} - \hat {P} _ {0 , 1})}} & \hat {v} _ {1} \in (0, 1), \\ + \infty & \hat {v} _ {1} = 1 \end{array} \right.,
$$
where  $\hat{q}_1 = \frac{\hat{P}_{1,1}}{\hat{P}_{0,1} + \hat{P}_{1,0}}$  and  $\hat{q}_0 = \frac{\hat{P}_{1,0}}{\hat{P}_{0,1} + \hat{P}_{1,0}}$  (See Claim 1 for the calculation of  $\hat{\mathbf{q}}$  from  $\hat{\mathbf{P}}$ ).
We naturally extend the process to non-binary settings. For example, review signals contain multiple grades  $(\Sigma = \{-2(\text{reject}), -1(\text{weak reject}), 1(\text{weak accept}), 2(\text{accept})\})$ .
Empirical Surprisal-based Score (General) When  $\hat{v}_s > 0\forall s\in \Sigma$  , we construct the prediction matrix  $\hat{\mathbf{P}}$  . The calibrated score is defined as
$$
\mathrm {S} \left(\left(\hat {x} _ {i}, \hat {\mathbf {p}} _ {i}\right) _ {i \in [ n ]}\right) = \left(\mathbb {E} _ {\hat {\mathbf {v}}} \varphi - \mathbb {E} _ {\hat {\mathbf {q}}} \varphi\right) \det  (\hat {\mathbf {U}}) ^ {- \frac {1}{2 (| \Sigma | - 1)}},
$$
where  $\forall s\in \Sigma ,\hat{q}_s = \left(\sum_t\frac{P_{s,t}}{\hat{P}_{t,s}}\right)^{-1}$ $(0 / 0\equiv 0)$ . When  $\operatorname *{det}(\hat{\mathbf{U}}) < 0$ $\mathrm{S}(\cdot)$  remains undefined. This "bad event" arises when a reviewer favors a paper, her belief in the likelihood of another reviewer also favoring it decreases. This unusual scenario alerts the Program Committee that this particular paper warrants further discussion and careful consideration before making a final decision. When comparing tasks with undefined scores, our method degenerates to the baseline, i.e., comparing the value of  $\mathbb{E}_{\hat{\mathbf{v}}}\varphi$
5 Theoretical Guarantee
In this section, we theoretically analyse the performance of our Empirical Surprisal-based Score. We use the error probability to measure the performance. Recall that we use the expected score of a paper in the clean state to measure its true quality. In the binary case, the true quality of a paper, whose clean state is  $\mathbf{w}$ , is  $w_{1}$ , i.e., the fraction of "accept" ratings in the clean setting. The error probability is the probability that the score ranks paper A higher than paper B while B's quality is better than A's.
Theorem 2 (binary) and Theorem 3 (general) show the theoretical upper bound of the error probability of our method. When the number of reviewers goes to infinity, the error probability goes to zero. The analysis follows from the results of Corollary 1 and a standard concentration bound analysis. We defer the proofs of these two theorems to Appendix C.
Theorem 2 (Error probability, binary case). For any pair of papers  $A, B$  whose clean states are  $\mathbf{w}^A, \mathbf{w}^B$  correspondingly (without loss of generality, let  $w_1^A < w_1^B$ ), given their noise matrices
$\mathbf{M}^A, \mathbf{M}^B \in \mathcal{M}^*$  correspondingly, the error probability  $\operatorname*{Pr}[\mathrm{S}(A) > \mathrm{S}(B)|w_1^A, w_1^B] + \frac{1}{2}\operatorname*{Pr}[\mathrm{S}(A) = \mathrm{S}(B)|w_1^A, w_1^B]$  goes to 0 when the number of reviewers  $n_A, n_B$  goes to infinity, and is bounded by
$$
\left(\hat {w} _ {1} ^ {A}\right) ^ {n _ {A}} + \left(1 - \hat {w} _ {1} ^ {B}\right) ^ {n _ {B}} - \frac {1}{2} \Big [ \left(\hat {w} _ {1} ^ {A}\right) ^ {n _ {A}} \left(\hat {w} _ {1} ^ {B}\right) ^ {n _ {B}} + \left(1 - \hat {w} _ {1} ^ {A}\right) ^ {n _ {A}} \left(1 - \hat {w} _ {1} ^ {B}\right) ^ {n _ {B}} \Big ] + \exp \bigg \{- \frac {2 \left(w _ {1} ^ {B} - w _ {1} ^ {A}\right) ^ {2}}{\frac {1}{n _ {A} (1 - \lambda_ {A}) ^ {2}} + \frac {1}{n _ {B} (1 - \lambda_ {B}) ^ {2}}} \bigg \},
$$
where  $\lambda_{A}$  is the noise level of paper  $A$  and  $\lambda_{B}$  is the noise level of paper  $B^{9}$ .
The error bound has three terms. The first two terms bound the error probability conditioning on at least one of the papers having an infinite score, i.e., every reviewer votes for "accept" ("reject"). The last term bounds the error probability conditioning on both papers has at least an "accept" and a "reject" vote. This theorem shows that we can achieve a better error bound when the rating difference  $(w_{1}^{B} - w_{1}^{A})$  between two papers is larger, when there are more reviewers ( $n_{A}$  and  $n_{B}$ ), and when the noise levels ( $\lambda_{A}$  and  $\lambda_{B}$ ) are lower.
Theorem 3 (Error probability, general case). For any pair of tasks  $A, B$  whose clean states are  $\mathbf{w}^A, \mathbf{w}^B$  correspondingly (without loss of generality, let  $\mathbb{E}_{\mathbf{w}^A} \varphi < \mathbb{E}_{\mathbf{w}^B} \varphi$ ), given their noise matrices  $\mathbf{M}^A, \mathbf{M}^B \in \mathcal{M}^*$  correspondingly, the error probability  $\operatorname*{Pr}\left[\mathrm{S}(A) > \mathrm{S}(B) \mid \mathbf{w}^A, \mathbf{w}^B\right] + \frac{1}{2} \operatorname*{Pr}\left[\mathrm{S}(A) = \mathrm{S}(B) \mid \mathbf{w}^A, \mathbf{w}^B\right]$  goes to 0 when the number of reviewers  $n_A, n_B$  goes to infinity, and is bounded by<sup>10</sup>
$$
\sum_ {s \in \Sigma} \left(1 - \hat {w} _ {s} ^ {A}\right) ^ {n _ {A}} + \sum_ {s \in \Sigma} \left(1 - \hat {w} _ {s} ^ {B}\right) ^ {n _ {B}} + \exp \left\{- \frac {2 \left(\mathbb {E} _ {\mathbf {w} ^ {B}} \varphi - \mathbb {E} _ {\mathbf {w} ^ {A}} \varphi\right) ^ {2}}{\left(\varphi_ {\operatorname* {m a x}} - \varphi_ {\operatorname* {m i n}}\right) ^ {2} \left(\frac {1}{n _ {A} (1 - \lambda_ {A}) ^ {2}} + \frac {1}{n _ {B} (1 - \lambda_ {B}) ^ {2}}\right)} \right\}.
$$
We perform numerical experiments to compare the performance of our Surprisal-based Score and the baseline score in the binary setting  $(\Sigma = \{0(\text{reject}), 1(\text{accept})\})$ . Recall that the baseline is the proportion of the "accept" ratings. Here we describe the parameters we select in numerical experiments.
1. The number of agents  $n$ : We perform the experiments in the settings of  $n = 3$  and  $n = 5$ .
2. The prior distribution of states  $Q$ : In the binary case, there exists a one-to-one mapping between the state  $\mathbf{w} = (w_0 = 1 - w_1, w_1)$  and  $w_1 \in [0,1]$ , which is the probability that a random reviewer votes for "accept" in the clean setting. The prior over the states can be described by a prior distribution over  $w_1$ . We use three different priors for  $w_1$ , which are  $\mathrm{Beta}\left(\frac{1}{2}, \frac{1}{2}\right)$  (most papers' quality is either high or low),  $\mathrm{Beta}(1,1)$  (papers' quality is distributed uniformly), and  $\mathrm{Beta}(3,3)$  (most papers have a medium quality).
3. The bias vector  $\mathbf{b}$ : Since the numerical experiment is to compare the scores of two papers, we consider the opposite and same biases between two papers:
- opposite: Paper A has the positive bias vector  $\mathbf{b}_A = [0,1]$  where the reviewers intend to vote for "accept" without careful review. Paper B has an negative bias vector  $\mathbf{b}_B = [1,0]$ . This simulates a situation where one paper has a negative cheap signal and another paper has a positive cheap signal.
- same: Both paper A and B have the same bias vector  $\mathbf{b}_A = \mathbf{b}_B = [0,1]^{11}$ . This simulates a situation where both papers have positive (negative) cheap signals.
We evaluate the performance by accuracy, which is 1 minus the expected error probability, and perform the experiments when two papers have opposite biases (Figure 4) and the same bias (Figure 5). In each setting, we consider three scenarios with varying noise levels for paper A ( $\lambda_{A} = 0,0.3,0.6$ ). In each scenario, we vary the noise level of paper B and compare our Surprisal-based Score and baseline under different priors and the number of reviewers. The x axis is paper B's noise level and the y axis is the accuracy of Surprisal-based Score (red lines) and baseline (green lines) $^{12}$ . According to the results, our method outperforms or equals the baseline in accuracy in all cases. We also notice that our method has more advantages under the opposite bias setting and when the number of reviewers is higher.
Note that the bound is independent of the biases.
$^{10}\varphi_{\mathrm{max}}$  and  $\varphi_{\mathrm{min}}$  are the abbreviations of  $\max_{s\in \Sigma}\varphi (s)$  and  $\min_{s\in \Sigma}\varphi (s)$  respectively.
<sup>11</sup>Since the prior we choose is symmetric, for the case of  $\mathbf{b}_A = \mathbf{b}_B = [1,0]$ , the result is exactly the same.
12The discontinuity happens because when  $\lambda_{A}$  is fixed and  $\lambda_{B}$  increases to a certain threshold, 1 "accept's" score can be higher than 2 "accept's" score.
In addition, we propose another benchmark for comparison, named the SP-inspired score. This benchmark slightly modifies the original Surprisingly Popular method [2] to fit our setting. This benchmark is not invariant to noise but usually has a lower variance. We include the details and the numerical experiments comparing our method with the SP-inspired score in Appendix D.
(a) 3 reviewers
Figure 4: Performance evaluation (opposite biases)
(b) 5 reviewers
Figure 5: Performance evaluation (same bias)
7 Conclusion and Discussion
When papers receive ratings of different noises due to heterogeneity of paper topics and the existence of cheap signals, we propose a scoring process that leads to a noise-robust rank among papers. The implementation of the process requires the reviewers to report their predictions for other ratings but does not require any prior knowledge. We provide theoretical and numerical justification for the process.
Our theory assumes that the reviewers are perfect Bayesian. However, this may not be the case in practice. Non-Bayesian reviewers may exhibit reporting bias and prediction bias. To some extent, prediction bias is inevitable even with a rational crowd, since it is hard for reviewers to fully understand the prior. The good news is, if the prediction bias is identically distributed among reviewers, then it can be regarded as part of the noisy prior, making the solution we propose still feasible. In contrast, dealing with reporting bias in the one-shot setting appears to be more challenging, since it can violet the basic assumption of homogeneous reviewers. For instance, some reviewers may be more lenient, while others are more strict. Many related works have considered this situation and proposed solutions to mitigate or calibrate the reporting error. We can employ any of these report debiasing schemes based on historical information or multi-tasking, and then use our Surprisal-based Score on the processed reports to measure the paper quality.
This research is supported by National Key R&D Program of China (2022ZD0114900).
[20] Sijun Tan, Jibang Wu, Xiaohui Bei, and Haifeng Xu. Least square calibration for peer reviews. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 27069-27080. Curran Associates, Inc., 2021.
[21] Hadi Hosseini, Debmalya Mandal, Nisarg Shah, and Kevin Shi. Surprisingly popular voting recovers rankings, surprisingly! In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 245–251. International Joint Conferences on Artificial Intelligence Organization, 8 2021. Main Track.
[22] Grant Schoenebeck and Biaoshuai Tao. Wisdom of the crowd voting: Truthful aggregation of voter information and preferences. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 1872-1883. Curran Associates, Inc., 2021.
[23] Grant Schoenebeck and Fang-Yi Yu. Two strongly truthful mechanisms for three heterogeneous agents answering one question. In Xujin Chen, Nikolai Gravin, Martin Hoefer, and Ruta Mehta, editors, Web and Internet Economics, pages 119-132, Cham, 2020. Springer International Publishing.
[24] David M Rothschild and Justin Wolfers. Forecasting elections: Voter intentions versus expectations. SSRN Electronic Journal, 2011.
[25] Anirban Dasgupta, Ravi Kumar, and D Sivakumar. Social sampling. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 235-243, 2012.
[26] Asa B Palley and Jack B Soll. Extracting the wisdom of crowds when information is shared. Management Science, 65(5):2291-2309, 2019.
[27] Marcellin Martinie, Tom Wilkening, and Piers DL Howe. Using meta-predictions to identify experts in the crowd when past performance is unknown. Plos one, 15(4):e0232058, 2020.
[28] Yi-Chun Chen, Manuel Mueller-Frank, and Mallesh M Pai. The wisdom of the crowd and higher-order beliefs. arXiv preprint arXiv:2102.02666, 2021.
[29] Tom Wilkening, Marcellin Martinie, and Piers DL Howe. Hidden experts in the crowd: Using meta-predictions to leverage expertise in single-question prediction problems. Management Science, 68(1):487-508, 2022.
[30] Asa B Palley and Ville A Satopää. Boosting the wisdom of crowds within a single judgment problem: Weighted averaging based on peer predictions. Management Science, 2023.
[31] Cem Peker. Extracting the collective wisdom in probabilistic judgments. Theory and Decision, 94(3):467-501, 2023.
[32] Robert L Winkler and Allan H Murphy. "good" probability assessors. Journal of Applied Meteorology and Climatology, 7(5):751-758, 1968.
[33] Yuqing Kong and Grant Schoenebeck. Equilibrium selection in information elicitation without verification via information monotonicity. In Anna R. Karlin, editor, 9th Innovations in Theoretical Computer Science Conference, ITCS 2018, January 11-14, 2018, Cambridge, MA, USA, volume 94 of LIPics, pages 13:1-13:20. Schloss Dagstuhl - Leibniz-Zentrum fuer Informatik, 2018.
A Detailed Examples of Our Model
Example 5 (Binary case, clean setting). Signal 0 represents "reject" and signal 1 represents "accept". There are three types of papers,  $\alpha$ ,  $\beta$ , and  $\gamma$ . Type  $\alpha$  paper's state is  $\mathbf{w}^{\alpha} = [w_0^{\alpha}, w_1^{\alpha}] = [0.8, 0.2]$ . That is, in the clean setting, a reviewer will receive "accept" signal for type  $\alpha$  paper with probability  $w_1^{\alpha} = 0.2$ . Analogously, we let  $w_1^{\beta} = 0.5$  and  $w_1^{\gamma} = 0.8$ . A random paper's state has type  $\alpha$ ,  $\beta$ , and  $\gamma$  with equal probability. This describes the prior distribution over paper's states  $Q$ :
$$
\mathbf {W} = \left\{ \begin{array}{l l} \mathbf {W} ^ {\alpha} & w. p. \frac {1}{3} \\ \mathbf {W} ^ {\beta} & w. p. \frac {1}{3} \\ \mathbf {W} ^ {\gamma} & w. p. \frac {1}{3} \end{array} \right..
$$
The prior probability that a reviewer receive "reject" signal for a random paper is  $q_0 = \frac{1}{3} w_0^\alpha + \frac{1}{3} w_0^\beta + \frac{1}{3} w_0^\gamma = 0.5, q_1 = 1 - q_0 = 0.5$ . The probability that both of the two reviewers receive "reject" for a paper will be  $U_{0,0} = \frac{1}{3} (w_0^\alpha)^2 + \frac{1}{3} (w_0^\beta)^2 + \frac{1}{3} (w_0^\gamma)^2 = 0.31$ . In general, the joint distribution between two reviewers' ratings is represented as a matrix:
$$
\begin{array}{l} \mathbf {U} = \frac {1}{3} \mathbf {U} ^ {\alpha} + \frac {1}{3} \mathbf {U} ^ {\beta} + \frac {1}{3} \mathbf {U} ^ {\gamma} \\ = \frac {1}{3} \left[ \begin{array}{c} w _ {0} ^ {\alpha} \\ w _ {1} ^ {\alpha} \end{array} \right] \left[ \begin{array}{c} w _ {0} ^ {\alpha}, w _ {1} ^ {\alpha} \end{array} \right] + \frac {1}{3} \left[ \begin{array}{c} w _ {0} ^ {\beta} \\ w _ {1} ^ {\beta} \end{array} \right] \left[ \begin{array}{c} w _ {0} ^ {\beta}, w _ {1} ^ {\beta} \end{array} \right] + \frac {1}{3} \left[ \begin{array}{c} w _ {0} ^ {\gamma} \\ w _ {1} ^ {\gamma} \end{array} \right] \left[ \begin{array}{c} w _ {0} ^ {\gamma}, w _ {1} ^ {\gamma} \end{array} \right] \\ = \frac {1}{3} \left[ \begin{array}{l} 0. 8 \\ 0. 2 \end{array} \right] [ 0. 8, 0. 2 ] + \frac {1}{3} \left[ \begin{array}{l} 0. 5 \\ 0. 5 \end{array} \right] [ 0. 5, 0. 5 ] + \frac {1}{3} \left[ \begin{array}{l} 0. 2 \\ 0. 8 \end{array} \right] [ 0. 2, 0. 8 ] \\ = \left[ \begin{array}{c c} 0. 3 1 & 0. 1 9 \\ 0. 1 9 & 0. 3 1 \end{array} \right]. \\ \end{array}
$$
According to the joint distribution matrix  $\mathbf{U}$ , the prediction matrix  $\mathbf{P}$  can be established:
$$
\mathbf {P} = \left[ \begin{array}{c c} \frac {U _ {0 , 0}}{U _ {0 , 0} + U _ {0 , 1}} & \frac {U _ {0 , 1}}{U _ {0 , 0} + U _ {0 , 1}} \\ \frac {U _ {1 , 0}}{U _ {1 , 0} + U _ {1 , 1}} & \frac {U _ {1 , 1}}{U _ {1 , 0} + U _ {1 , 1}} \end{array} \right] = \left[ \begin{array}{c c} 0. 6 2 & 0. 3 8 \\ 0. 3 8 & 0. 6 2 \end{array} \right].
$$
Thus, when a reviewer receives "reject" signal, she will believe a random reviewer will receive "accept" signal with probability  $P_{0,1} = 0.38$ . When a reviewer receives "accept" signal, she will believe a random reviewer will receive "accept" signal with probability  $P_{1,1} = 0.62$ .
Example 6 (Binary case, noisy setting). We use the same prior in Example 5. Given a paper of type  $\gamma$  (i.e. the probability that the clean signal of a reviewer is "accept" is 0.8), the paper is difficult to read so that there is a  $50\%$  chance that a reviewer will vote for "reject" without reading it carefully. This shows that the paper has a noise matrix  $\mathbf{M} = \begin{bmatrix} 1 & 0 \\ 0.5 & 0.5 \end{bmatrix}$ . When the noise applies,  $\forall s, t \in \Sigma, \hat{U}_{s,t} = \sum_{a,b} U_{a,b} M_{a,s} M_{b,t}$ . Thus,  $\hat{\mathbf{U}} = \mathbf{M}^{\top} \mathbf{U} \mathbf{M}$ , thereby we obtain
$$
\hat {\mathbf {U}} = \mathbf {M} ^ {\top} \mathbf {U} \mathbf {M} \approx \left[ \begin{array}{c c} 0. 5 8 & 0. 1 7 \\ 0. 1 7 & 0. 0 8 \end{array} \right],
$$