SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Because each  $x_{i}^{A}$  and  $x_{j}^{B}$  is independent, we can define independent random variables  $y_{k}\big(k\in [1,n^{A} + n^{B}]\big)$  as
$$
y _ {k} = \left\{ \begin{array}{l l} \frac {\varphi (\hat {x} _ {k} ^ {A})}{n ^ {A} (1 - \lambda^ {A})} & k \leq n ^ {A} \\ \frac {- \varphi (\hat {x} _ {k - n A} ^ {B})}{n ^ {B} (1 - \lambda^ {B})} & k > n ^ {A} \end{array} \right..
$$
For  $k \in [1, n^A]$ ,  $\frac{\varphi_{\min}}{n^A(1 - \lambda^A)} \leq y_k \leq \frac{\varphi_{\max}}{n^A(1 - \lambda^A)}$ . For  $k \in [n^A + 1, n^A + n^B]$ ,  $-\frac{\varphi_{\max}}{n^B(1 - \lambda^B)} \leq y_k \leq -\frac{\varphi_{\min}}{n^B(1 - \lambda^B)}$ . Using the variables  $y_k (k \in [1, n^A + n^B])$ , we can apply the standard Hoeffding's Inequality.
$$
\begin{array}{l} \Pr \left[ \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {A}} \varphi - \mathbb {E} _ {\hat {\mathbf {q}} ^ {A}} \varphi}{\operatorname* {d e t} (\hat {\mathbf {U}} ^ {A}) ^ {\frac {1}{2 (| \Sigma | - 1)}}} \geq \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {B}} \varphi - \mathbb {E} _ {\hat {\mathbf {q}} ^ {B}} \varphi}{\operatorname* {d e t} (\hat {\mathbf {U}} ^ {B}) ^ {\frac {1}{2 (| \Sigma | - 1)}}} \left| \mathbf {w} ^ {A}, \mathbf {w} ^ {B} \right. \right] \\ = \mathrm {P r} \left[ \sum_ {k = 1} ^ {n ^ {A} + n ^ {B}} y _ {k} - \mathbb {E} \left[ \sum_ {k = 1} ^ {n ^ {A} + n ^ {B}} y _ {k} \right] \geq \mathbb {E} _ {\mathbf {w} ^ {B}} \varphi - \mathbb {E} _ {\mathbf {w} ^ {A}} \varphi | \mathbf {w} ^ {A}, \mathbf {w} ^ {B} \right] \\ \leq \exp \left\{- \frac {2 \left(\mathbb {E} _ {\mathbf {w} ^ {B}} \varphi - \mathbb {E} _ {\mathbf {w} ^ {A}} \varphi\right) ^ {2}}{n ^ {A} \left(\frac {\varphi_ {\operatorname* {m a x}} - \varphi_ {\operatorname* {m i n}}}{n ^ {A} (1 - \lambda^ {A})}\right) ^ {2} + n ^ {B} \left(\frac {\varphi_ {\operatorname* {m a x}} - \varphi_ {\operatorname* {m i n}}}{n ^ {B} (1 - \lambda^ {B})}\right) ^ {2}} \right\} \quad (\text {H o e f f d i n g ' s I n e q u a l i t y}) \\ = \exp \left\{- \frac {2 \left(\mathbb {E} _ {\mathbf {w} ^ {B}} \varphi - \mathbb {E} _ {\mathbf {w} ^ {A}} \varphi\right) ^ {2}}{\left(\varphi_ {\mathrm {m a x}} - \varphi_ {\mathrm {m i n}}\right) ^ {2} \left(\frac {1}{n ^ {A} (1 - \lambda^ {A}) ^ {2}} + \frac {1}{n ^ {B} (1 - \lambda^ {B}) ^ {2}}\right)} \right\} \\ \end{array}
$$
Theorem 2 (Error probability, binary case). For any pair of papers  $A, B$  whose clean states are  $\mathbf{w}^A, \mathbf{w}^B$  correspondingly (without loss of generality, let  $w_1^A < w_1^B$ ), given their noise matrices  $\mathbf{M}^A, \mathbf{M}^B \in \mathcal{M}^*$  correspondingly, the error probability  $\operatorname{Pr}\left[\mathrm{S}(A) > \mathrm{S}(B) | w_1^A, w_1^B\right] + \frac{1}{2} \operatorname{Pr}\left[\mathrm{S}(A) = \mathrm{S}(B) | w_1^A, w_1^B\right]$  goes to 0 when the number of reviewers  $n_A, n_B$  goes to infinity, and is bounded by
$$
\left(\hat {w} _ {1} ^ {A}\right) ^ {n _ {A}} + \left(1 - \hat {w} _ {1} ^ {B}\right) ^ {n _ {B}} - \frac {1}{2} \left[ \left(\hat {w} _ {1} ^ {A}\right) ^ {n _ {A}} \left(\hat {w} _ {1} ^ {B}\right) ^ {n _ {B}} + \left(1 - \hat {w} _ {1} ^ {A}\right) ^ {n _ {A}} \left(1 - \hat {w} _ {1} ^ {B}\right) ^ {n _ {B}} \right] + \exp \left\{- \frac {2 \left(w _ {1} ^ {B} - w _ {1} ^ {A}\right) ^ {2}}{\frac {1}{n _ {A} (1 - \lambda_ {A}) ^ {2}} + \frac {1}{n _ {B} (1 - \lambda_ {B}) ^ {2}}} \right\},
$$
where  $\lambda_{A}$  is the noise level of paper  $A$  and  $\lambda_{B}$  is the noise level of paper  $B^{14}$ .
Proof of Theorem 2. We will show that with high probability, the prediction matrix can be constructed. Once it is constructed, we use the result of Lemma 1.
14Note that the bound is independent of the biases.
Theorem 3 (Error probability, general case). For any pair of tasks  $A, B$  whose clean states are  $\mathbf{w}^A, \mathbf{w}^B$  correspondingly (without loss of generality, let  $\mathbb{E}_{\mathbf{w}^A} \varphi < \mathbb{E}_{\mathbf{w}^B} \varphi$ ), given their noise matrices  $\mathbf{M}^A, \mathbf{M}^B \in \mathcal{M}^*$  correspondingly, the error probability  $\operatorname*{Pr}\left[\mathrm{S}(A) > \mathrm{S}(B) \mid \mathbf{w}^A, \mathbf{w}^B\right] + \frac{1}{2} \operatorname*{Pr}\left[\mathrm{S}(A) = \mathrm{S}(B) \mid \mathbf{w}^A, \mathbf{w}^B\right]$  goes to 0 when the number of reviewers  $n_A, n_B$  goes to infinity, and is bounded by<sup>15</sup>
$$
\sum_ {s \in \Sigma} \left(1 - \hat {w} _ {s} ^ {A}\right) ^ {n _ {A}} + \sum_ {s \in \Sigma} \left(1 - \hat {w} _ {s} ^ {B}\right) ^ {n _ {B}} + \exp \left\{- \frac {2 \left(\mathbb {E} _ {\mathbf {w} ^ {B}} \varphi - \mathbb {E} _ {\mathbf {w} ^ {A}} \varphi\right) ^ {2}}{\left(\varphi_ {\max } - \varphi_ {\min }\right) ^ {2} \left(\frac {1}{n _ {A} (1 - \lambda_ {A}) ^ {2}} + \frac {1}{n _ {B} (1 - \lambda_ {B}) ^ {2}}\right)} \right\}.
$$
Proof of Theorem 3. First, we give an upper bound of the probability that the score is undefined.
$$
\begin{array}{l} \Pr [ S (A) \text {i s u n d e f i n e d} ] \\ = \Pr \left[ \exists s \in \Sigma , \forall i \in \left[ n ^ {A} \right], \hat {x} _ {i} \neq s \right] \\ \leq \sum_ {s \in \Sigma} \Pr \left[ \forall i \in \left[ n ^ {A} \right], \hat {x} _ {i} \neq s \right] \\ = \sum_ {s \in \Sigma} \left(1 - \hat {w} _ {s} ^ {A}\right) ^ {n ^ {A}} \\ \end{array}
$$
Similarly,  $\operatorname*{Pr}[S(B)$  is undefined]  $\leq \sum_{s\in \Sigma}\left(1 - \hat{w}_s^B\right)^n$  . We have proved that with high probability, the prediction matrix can be constructed thus the scores are not undefined. We then use the result
$^{15}\varphi_{\mathrm{max}}$  and  $\varphi_{\mathrm{min}}$  are the abbreviations of  $\max_{s\in \Sigma}\varphi (s)$  and  $\min_{s\in \Sigma}\varphi (s)$  respectively.
of Lemma 1.
$$
\begin{array}{l} \Pr [ \mathrm {S} (A) > \mathrm {S} (B) | w _ {1} ^ {A}, w _ {1} ^ {B} ] + \frac {1}{2} \Pr [ \mathrm {S} (A) = \mathrm {S} (B) | w _ {1} ^ {A}, w _ {1} ^ {B} ] \\ \leq \Pr [ S (A) \text {i s u n d e f i n e d} ] + \Pr [ S (B) \text {i s u n d e f i n e d} ] \\ + \Pr [ S (A) \geq S (B) \cap \text {b o t h} S (A) \text {a n d} S (B) \text {a r e n o t u n d e f i n e d} | w _ {1} ^ {A}, w _ {1} ^ {B} ] \\ \leq \sum_ {s \in \Sigma} \left(1 - \hat {w} _ {s} ^ {A}\right) ^ {n ^ {A}} + \sum_ {s \in \Sigma} \left(1 - \hat {w} _ {s} ^ {B}\right) ^ {n ^ {B}} \\ + \Pr \left[ \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {A}} \varphi - \mathbb {E} _ {\hat {\mathbf {q}} ^ {A}} \varphi}{\det  (\hat {\mathbf {U}} ^ {A}) ^ {\frac {1}{2 (| \Sigma | - 1)}}} \geq \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {B}} \varphi - \mathbb {E} _ {\hat {\mathbf {q}} ^ {B}} \varphi}{\det  (\hat {\mathbf {U}} ^ {B}) ^ {\frac {1}{2 (| \Sigma | - 1)}}} \left| \mathbf {w} ^ {A}, \mathbf {w} ^ {B} \right] \right. \\ \leq \sum_ {s \in \Sigma} \left(1 - \hat {w} _ {s} ^ {A}\right) ^ {n ^ {A}} + \sum_ {s \in \Sigma} \left(1 - \hat {w} _ {s} ^ {B}\right) ^ {n ^ {B}} \\ + \exp \left\{- \frac {2 \left(\mathbb {E} _ {\mathbf {w} ^ {B}} \varphi - \mathbb {E} _ {\mathbf {w} ^ {A}} \varphi\right) ^ {2}}{\left(\varphi_ {\max } - \varphi_ {\min }\right) ^ {2} \left(\frac {1}{n ^ {A} (1 - \lambda^ {A}) ^ {2}} + \frac {1}{n ^ {B} (1 - \lambda^ {B}) ^ {2}}\right)} \right\} \tag {Lemma 1} \\ \end{array}
$$
D SP-inspired Score
In this section, we introduce a new benchmark for comparison, inspired by the Surprising Popular (SP) mechanism, which we term as the SP-inspired score. The Surprising Popular mechanism is designed to consolidate multiple reports into a single assertion of which world state we are in, rather than generating scores to compare different alternatives. Utilizing the notations explained earlier in this paper, mapping  $\varphi : \Sigma \to \mathbb{R}$  represents each signal's quality,  $\hat{v}_s = \frac{1}{n} \sum_{i=1}^{n} \mathbf{1}[\hat{x}_i = s]$  stands for the frequency of the noisy signal  $s$ , and  $\hat{q}_s = \left( \sum_t \frac{\hat{P}_{s,t}}{\hat{P}_{t,s}} \right)^{-1}$  represents the reconstructed prior probability of the noisy signal  $s$ . When dealing with binary signals ( $\Sigma = \{0,1\}$ ), the SP mechanism presumes that the true state is 1 if  $\frac{\hat{v}_1}{\hat{q}_1} > \frac{\hat{v}_0}{\hat{q}_0}$ ; otherwise, it is presumed that the true state is 0. In general settings, the SP mechanism determines the true state as the signal  $s$  that maximizes  $\frac{\hat{v}_s}{\hat{q}_s}$ .
To facilitate comparisons between alternatives, we define the SP-inspired score as
$$
\mathrm {S} \big ((\hat {x} _ {i}, \hat {\mathbf {p}} _ {i}) _ {i \in [ n ]} \big) = \sum_ {s \in \Sigma} \varphi (s) \frac {\hat {v} _ {s}}{\hat {q} _ {s}}.
$$
In the case of binary signals, we set  $\varphi(0) = -1$  and  $\varphi(1) = 1$ . This results in the SP-inspired score being  $S((\hat{x}_i, \hat{\mathbf{p}} i) i \in [n]) = \frac{\hat{v}_1}{\hat{q}_1} - \frac{\hat{v}_0}{\hat{q}_0}$ .
Difference between Surprisal-based Score and SP-inspired Score Recall our Empirical Surprisal-based Score is defined as
$$
\mathrm {S} \left(\left(\hat {x} _ {i}, \hat {\mathbf {p}} _ {i}\right) _ {i \in [ n ]}\right) = \left(\mathbb {E} _ {\hat {\mathbf {q}}} \varphi - \mathbb {E} _ {\hat {\mathbf {q}}} \varphi\right) \det  (\hat {\mathbf {U}}) ^ {- \frac {1}{2 (| 2 | - 1)}}.
$$
Both our Surprisal-based Score and the SP-inspired score involve reconstructing the prior from agents' predictions. However, they differ in how to exploit this reconstructed prior to obtain a calibrated score. Our Surprisal-based Score is invariant of noise, meaning it does not change in the presence of noise when the number of reviewers is infinite. On the other hand, the SP-inspired score is affected by noise. This implies that in certain situations, even when we have an infinite number of reviewers, the SP-inspired score fails to distinguish which of the two papers is better. However, our Surprisal-based Score exhibits lower stability compared to the SP-inspired score, because it involves a division of the correlation, which may lead to a high variance when the number of reviewers is small.
Numerical Experiments We perform numerical experiments to compare the performance of our Surprisal-based Score and the SP-inspired score in the binary setting  $(\Sigma = \{0(\text{reject}), 1(\text{accept})\})$ . We employ the same experimental setup as described in Section 6. Figure 6 and Figure 7 present the experimental results. Upon analyzing the results, we observed that the performance of our Surprisal-based Score and the SP-inspired Score is roughly the same when the number of reviewers is small.
Figure 6: Performance evaluation (opposite biases)
Figure 7: Performance evaluation (same bias)