SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\hat {\mathbf {P}} = \left[ \begin{array}{c c} \frac {\hat {U} _ {0 , 0}}{\hat {U} _ {0 , 0} + \hat {U} _ {0 , 1}} & \frac {\hat {U} _ {0 , 1}}{\hat {U} _ {0 , 0} + \hat {U} _ {0 , 1}} \\ \frac {\hat {U} _ {1 , 0}}{\hat {U} _ {1 , 0} + \hat {U} _ {1 , 1}} & \frac {\hat {U} _ {1 , 1}}{\hat {U} _ {1 , 0} + \hat {U} _ {1 , 1}} \end{array} \right] \approx \left[ \begin{array}{c c} 0. 7 7 & 0. 2 3 \\ 0. 6 8 & 0. 3 2 \end{array} \right].
$$
If a reviewer receives and votes for "reject", she will predict a random reviewer will receive and vote for "accept" with probability  $\hat{P}_{0,1} = 0.23$ . If a reviewer receives and votes for "accept", she will predict a random reviewer will receive and vote for "accept" with probability  $\hat{P}_{1,1} = 0.32$ .
B Invariant Surprisal
We introduce a measure that is invariant to any invertible noise.
Definition 4 (Invariant Surprisal). Given state  $\mathbf{w}$  and joint distribution  $\mathbf{U}$ , we define the invariant surprisal as
$$
\operatorname {S u r p} (\mathbf {w}, \mathbf {U}) = (\mathbf {w} - \mathbf {q}) \mathbf {U} ^ {- 1} \left(\mathbf {w} ^ {\top} - \mathbf {q} ^ {\top}\right)
$$
where prior  $\mathbf{q}$  is the marginal distribution of  $\hat{\mathbf{U}}$ .
The invariant surprisal is closely related to the affine-invariant Mahalanobis distance, which measures the distance between a point  $\mathbf{x}$  and a distribution  $D$  by  $d_{M}(\mathbf{x}, D) = \sqrt{(\mathbf{x} - \boldsymbol{\mu})\mathbf{S}^{-1}(\mathbf{x} - \boldsymbol{\mu})}$  where
$\pmb{\mu} = \mathbb{E}[\mathbf{x}]$  and  $\mathbf{S}$  is the covariance matrix. The invariant surplasal is closely related to  $d_{M}(\mathbf{w},Q)^{2}$ . However, we cannot directly use  $d_{M}(\mathbf{w},Q)^{2}$ . In our setting,  $\mathbf{w}$  is a distribution vector where the sum of the elements is 1, thus the covariance matrix  $\mathbf{S}$  is degenerate<sup>13</sup> and  $\mathbf{S}^{-1}$  is not well-defined. Thus, we modify  $d_{M}(\mathbf{w},Q)^{2}$  by using the simple joint distribution matrix  $\mathbf{U}$  rather than the covariance matrix. It is invariant and well-defined. When the number of distinct states is  $|\Sigma|$ , it matches Shannon's surprisal to some extent.
Theorem 4. The invariant surprisal is non-negative, vanishes when the state  $\mathbf{w} = \mathbf{q}$  equals the prior, and is invariant to any invertible noise. In the binary case where  $\Sigma = \{0,1\}$ , the invariant surprisal is  $\mathrm{Surp}(\mathbf{w},\mathbf{U}) = (\det (\mathbf{U}))^{-1}(w_1 - q_1)^2 = (q_0q_1(P_{1,1} - P_{0,1}))^{-1}(w_1 - q_1)^2$  and  $q_0 = \frac{P_{1,0}}{P_{0,1} + P_{1,0}}$ ,  $q_1 = \frac{P_{0,1}}{P_{0,1} + P_{1,0}}$ .
Proof of Theorem 4.
$$
\begin{array}{l} (\hat {\mathbf {w}} - \hat {\mathbf {q}}) \hat {\mathbf {U}} ^ {- 1} (\hat {\mathbf {w}} - \hat {\mathbf {q}}) ^ {\top} \\ = (\mathbf {w} - \mathbf {q}) \mathbf {M} (\mathbf {M} ^ {\top} \mathbf {U} \mathbf {M}) ^ {- 1} \mathbf {M} ^ {\top} (\mathbf {w} - \mathbf {q}) ^ {\top} \\ = (\mathbf {w} - \mathbf {q}) \mathbf {U} ^ {- 1} (\mathbf {w} - \mathbf {q}) ^ {\top} \\ \end{array}
$$
In the binary case where  $\Sigma = \{\text{reject,accept}\}$ , recall that we use 1 for accept and 0 for reject for short. Then, we can simplify the expression of surprisal in the binary case as follow.
$$
\begin{array}{l} \operatorname {S u r p} (\mathbf {w}, \mathbf {U}) \\ = (\mathbf {w} - \mathbf {q}) \mathbf {U} ^ {- 1} (\mathbf {w} ^ {\top} - \mathbf {q} ^ {\top}) \\ = \left[ \begin{array}{l l} w _ {0} - q _ {0} & w _ {1} - q _ {1} \end{array} \right] \frac {1}{\operatorname* {d e t} (\mathbf {U})} \left[ \begin{array}{c c} U _ {1, 1} & - U _ {0, 1} \\ - U _ {1, 0} & U _ {0, 0} \end{array} \right] \left[ \begin{array}{l} w _ {0} - q _ {0} \\ w _ {1} - q _ {1} \end{array} \right] \\ = \frac {1}{\operatorname* {d e t} (\mathbf {U})} (w _ {0} - q _ {0}) ^ {2} (\sum_ {s, t \in \Sigma} U _ {s, t}) \quad (w _ {0} - q _ {0} = - (w _ {1} - q _ {1})) \\ = \frac {1}{\det (\mathbf {U})} \left(w _ {0} - q _ {0}\right) ^ {2} \\ = \left(q _ {0} q _ {1} \left(P _ {0, 0} - P _ {1, 0}\right)\right) ^ {- 1} \left(w _ {0} - q _ {0}\right) ^ {2} \\ = \left(q _ {0} q _ {1} \left(P _ {1, 1} - P _ {0, 1}\right)\right) ^ {- 1} \left(w _ {1} - q _ {1}\right) ^ {2} \\ \end{array}
$$
The invariant surprisal is invariant to any invertible noise. However, it only reflects the distance between the state and the prior and is not suitable for many application scenarios like peer review. Nevertheless, we show that when people reach a consensus in the clean state, the invariant surprisal can be used to identify the clean state. Moreover, in this case, the invariant surprisal matches Shannon's surprisal to some degree.
Shannon's surprisal For a random variable  $X$  where  $\forall x, \operatorname{Pr}[X = x] = q_x$ , Shannon's definition of the surprisal score of the outcome  $x$  is  $\log \frac{1}{q_x}$ . When the outcome is more likely to happen, the surprisal that it actually happens is small. To measure the surprisal of a state  $\mathbf{w}$ , we can use  $\log \frac{1}{q_{\mathbf{w}}}$  where  $q_{\mathbf{w}}$  denotes the probability that the state  $\mathbf{w}$  shows up.
Example 7 (The consensus case: Invariant Surprisal  $\approx$  Surprisal). We consider a special case, call it the consensus case, where agents always have the same rating. Formally there are  $|\Sigma|$  distinct states and each state is a one-hot vector. For all  $s$ , the  $s^{th}$  state's coordinates are all zero except that the  $s^{th}$  coordinate is 1, denoted by  $\mathbf{1}_s$ . For example, when  $d = 3$ , the states are  $(1,0,0),(0,1,0),(0,0,1)$ . In this case,  $q_s$  is not only the prior probability for signal  $s$ , but also the probability that the  $s^{th}$  state shows up. Let  $\mathrm{diag}(\mathbf{q})$  denote a diagonal matrix whose diagonal elements consist of vector
For example,  $\mathbf{w}$  is either  $(1,0)$  or  $(0,1)$  with equal probability.  $\mathbf{S} = \begin{bmatrix} 0.25 & -0.25 \\ -0.25 & 0.25 \end{bmatrix}$
q.  $\mathbf{U} = \mathrm{diag}(\mathbf{q})$ . The invariant surprisal for the  $s^{th}$  state is
$$
\begin{array}{l} \operatorname {S u r p} \left(\mathbf {1} _ {s}, \mathbf {U}\right) \\ = (\mathbf {1} _ {s} - \mathbf {q}) \operatorname {d i a g} (\mathbf {q}) ^ {- 1} \left(\mathbf {1} _ {s} ^ {\top} - \mathbf {q} ^ {\top}\right) \\ = \frac {1}{q _ {s}} - 2 + 1 \\ = \frac {1}{q _ {s}} - 1. \\ \end{array}
$$
In the consensus case, there is a one-to-one correspondence between states and signals. Our definition for the surprisal of the state  $\mathbf{1}_s$  where all agents agree to signal  $s$  is  $\frac{1}{q_s} - 1 = \frac{1}{q_{\mathbf{1}_s}} - 1$  which is the 1st Taylor expansion of Shannon's.
In a special case where there are only  $|\Sigma|$  distinct states, we have the following result.
Corollary 2 (The noisy consensus case). When there are only  $|\Sigma|$  distinct states  $\mathbf{w}_1, \mathbf{w}_2, \dots, \mathbf{w}_{|\Sigma|}$ , the invariant surprisal of each state  $\mathbf{w}_k$  is the 1st Taylor expansion of Shannon's surprisal, i.e.,  $\mathrm{Surp}(\mathbf{w}_k, \mathbf{P}) = \frac{1}{q_{\mathbf{w}_k}} - 1$ .
We will show that the above case is a noisy version of the consensus case. The invariance directly induces the above result.
Proof of Corollary 2. When there are only  $|\Sigma|$  distinct states, we can define the intrinsic noise  $\mathbf{M}$  as a matrix where each  $k^{th}$  row is  $\mathbf{w}_k$  such that it is the noisy version of the consensus case. Then the invariance of the invariant surprisal leads to the above corollary.
Identifying Clean State by the Invariant Surprisal Given that agents reach consensus in the clean states, if we have already known the prior  $\mathbf{q}$  and the prior is unbalanced  $q_{s} \neq q_{t}, \forall s \neq t$ , then we can identify the clean state in the noisy setting by the invariant surprisal. Sometimes even side information about  $\mathbf{q}$  is sufficient. For example, in the binary case, if we know that the fraction of all "accept" is less than .5, i.e.,  $q_{a} < 0.5$ , then once we obtain  $>1$  invariant surprisal in the noisy setting, we know the clean state is all "accept", otherwise, the clean state is all "reject".
C.1 Proof of Claim 2
Proof of Claim 2. First, we show that  $\hat{\mathbf{w}} = (1 - \lambda)\mathbf{w} + \lambda \mathbf{b}$
$\hat{\mathbf{w}} = \mathbf{w}\mathbf{M}_{\lambda ,\mathbf{b}}$  (see Claim 5 for more details)
$= \mathbf{w}((1 - \lambda)\mathbf{I} + \lambda \mathbf{B})$  (Definition of  $\mathbf{M}_{\lambda ,\mathbf{b}}$
$= (1 - \lambda)\mathbf{w} + \lambda \mathbf{b}$
Then, we prove that every binary  $\mathbf{M} = \mathbf{M}_{\lambda ,\mathbf{b}}\in \mathcal{M}^*$  is invertible and satisfies  $M_{1,1} > M_{0,1}$ .
$$
\mathbf {M} _ {\lambda , \mathbf {b}} = (1 - \lambda) \mathbf {I} + \lambda \mathbf {B} = \left[ \begin{array}{c c} 1 - \lambda + \lambda b _ {0} & \lambda b _ {1} \\ \lambda b _ {0} & 1 - \lambda + \lambda b _ {1} \end{array} \right]
$$
$$
\det  (M) = M _ {0, 0} M _ {1, 1} - M _ {0, 1} M _ {1, 0} = 1 - \lambda \neq 0
$$
$$
M _ {1, 1} - M _ {0, 1} = 1 - \lambda > 0
$$
Finally, we prove that every invertible binary noise matrix  $\mathbf{M}$  with  $M_{1,1} > M_{0,1}$  belongs to  $\mathcal{M}^*$ .
$$
\begin{array}{l} \mathbf {M} = \left[ \begin{array}{c c} 1 - M _ {0, 1} & M _ {0, 1} \\ 1 - M _ {1, 1} & M _ {1, 1} \end{array} \right] \\ = \left(M _ {1, 1} - M _ {0, 1}\right) \mathbf {I} + \left(1 - M _ {1, 1} + M _ {0, 1}\right) \left[ \begin{array}{l l} 1 - \frac {M _ {0 , 1}}{1 - M _ {1 , 1} + M _ {0 , 1}} & \frac {M _ {0 , 1}}{1 - M _ {1 , 1} + M _ {0 , 1}} \\ 1 - \frac {M _ {0 , 1}}{1 - M _ {1 , 1} + M _ {0 , 1}} & \frac {M _ {0 , 1}}{1 - M _ {1 , 1} + M _ {0 , 1}} \end{array} \right] \\ \end{array}
$$
Let  $\lambda = 1 - M_{1,1} + M_{0,1}$  and  $\mathbf{b} = \left[1 - \frac{M_{0,1}}{\lambda},\frac{M_{0,1}}{\lambda}\right]$ . Then, we have  $\mathbf{M} = (1 - \lambda)\mathbf{I} + \lambda \mathbf{B}$ . Because
$M_{1,1} - M_{0,1} = \operatorname*{det}\bigl (\mathbf{M}\bigr)\neq 0$  and  $0\leq M_{0,1} < M_{1,1}\leq 1$ , we derive that  $\lambda \in [0,1)$ . Because  $1 - M_{1,1} > 0$ , we obtain that  $0 < \frac{M_{0,1}}{1 - M_{1,1} + M_{0,1}} < 1$ . Thus,  $\mathbf{b}\in \Delta_{\Sigma}$ .
C.2 Proof of Claim 3
Proof of Claim 3.
$$
\begin{array}{l} \det  (\mathbf {U}) = U _ {0, 0} U _ {1, 1} - U _ {0, 1} U _ {1, 0} \\ = q _ {0} q _ {1} \left(P _ {0, 0} P _ {1, 1} - P _ {0, 1} P _ {1, 0}\right) \\ = q _ {0} q _ {1} \left(\left(1 - P _ {0, 1}\right) P _ {1, 1} - P _ {0, 1} \left(1 - P _ {1, 1}\right)\right) \\ = q _ {0} q _ {1} \left(P _ {1, 1} - P _ {0, 1}\right). \\ \end{array}
$$
By plugging in the above formula into the definition of  $\mathrm{Surp}^*$ , we prove the claim.
Proof of Theorem 1. We first show that  $\mathbf{U}^{-1}$  is positive semi-definite, which implies the non-negativity. Afterwards, we prove the invariance.
Claim 4.  $\mathbf{U}$  and  $\mathbf{U}^{-1}$  are positive semi-definite.
Proof of Claim 4. As the number of papers is finite, we denote all  $K$  states as  $\mathbf{w}_1,\mathbf{w}_2,\dots ,\mathbf{w}_K$ , where state  $\mathbf{w}_i$  represents the probability distribution of a random reviewer's signal for paper  $i$  under review. Let  $1\times K$  vector  $\mathbf{q}_{\mathrm{state}}$  denote the prior distribution over the states. For all  $s,t$ ,  $U_{s,t} = \sum_{k}q_{\mathbf{w}_{k}}\mathbf{w}_{k}(s)\mathbf{w}_{k}(t)$ . Let  $\mathbf{W}$  be a  $K\times |\Sigma|$  matrix that stacks the states, i.e., for all  $k$ , the  $k^{th}$  row of  $\mathbf{W}$  is  $\mathbf{w}_k$ . The above equation is equivalent to  $\mathbf{U} = \mathbf{W}^{\top}\mathrm{diag}(\mathbf{q}_{\mathrm{state}})\mathbf{W}$  thus  $\mathbf{U}$  is positive semi-definite. Because a positive semi-definite matrix's inverse is also positive semi-definite,  $\mathbf{U}^{-1}$  is also positive semi-definite.
Claim 5.  $\hat{\mathbf{q}} = \mathbf{q}\mathbf{M},\hat{\mathbf{w}} = \mathbf{w}\mathbf{M},\hat{\mathbf{U}} = \mathbf{M}^{\top}\mathbf{U}\mathbf{M}.$
Proof of Claim 5. When noise  $\mathbf{M}$  applies,  $\forall t\in \Sigma ,\hat{w}_t = \sum_s w_s M_{s,t},\forall s,t\in \Sigma ,\hat{U}_{s,t} = \sum_{a,b}U_{a,b}M_{a,s}M_{b,t}$ . Thus,  $\hat{\mathbf{w}} = \mathbf{w}\mathbf{M}$ ,  $\hat{\mathbf{U}} = \mathbf{M}^{\top}\mathbf{U}\mathbf{M}$ . The noisy prior is the marginal distribution of the noisy joint distribution. Let  $\mathbf{1}^{1\times |\Sigma|}$  denote a  $|\Sigma|$ -dimensional row vector with all elements equal to one. We have
$$
\begin{array}{l} \hat {\mathbf {q}} = \mathbf {1} ^ {1 \times | \Sigma |} \hat {\mathbf {U}} \\ = \mathbf {1} ^ {1 \times | \Sigma |} \mathbf {M} ^ {\top} \mathbf {U} \mathbf {M} \quad (\mathbf {M} ^ {\top} \text {i s c o l u m n - s t o c h a s t i c}) \\ = \mathbf {1} ^ {1 \times | \Sigma |} \mathbf {U M} \\ = q M \\ \end{array}
$$
Claim 6.  $\operatorname*{det}(\mathbf{M}_{\lambda ,\mathbf{b}}) = (1 - \lambda)^{|\Sigma | - 1}$
Proof of Claim 6.
$$
\begin{array}{l} \det  (\mathbf {M} _ {\lambda , \mathbf {b}}) = \det  ((1 - \lambda) \mathbf {I} + \lambda \mathbf {B}) \\ = (1 - \lambda) ^ {| \Sigma |} \det  (\mathbf {I} + \frac {\lambda}{1 - \lambda} \mathbf {B}) \quad (\det  (c \mathbf {A}) = c ^ {n} \det  (\mathbf {A}) \text {w h e r e} \mathbf {A} \text {i s} n \times n \text {m a t r i x}) \\ = (1 - \lambda) ^ {| \Sigma |} \det  \left(\mathbf {I} + \frac {\lambda}{1 - \lambda} \mathbf {1} ^ {\top} \mathbf {b}\right) \quad (\mathbf {1} = (1, 1, \dots , 1)) \\ = (1 - \lambda) ^ {| \Sigma |} \left(1 + \frac {\lambda}{1 - \lambda} \mathbf {b} \mathbf {1} ^ {\top}\right) \quad (\text {S y l v e s t e r ' s d e t e r m i n a n t t h e o r e m}) \\ = (1 - \lambda) ^ {| \Sigma |} \left(1 + \frac {\lambda}{1 - \lambda}\right) \quad (\mathbf {b} \text {i s a d i s t r i b u t i o n v e c t o r t h u s a l l e m e n t s s u m t o 1 .}) \\ = (1 - \lambda) ^ {| \Sigma | - 1} \\ \end{array}
$$
Let  $\hat{\mathbf{w}},\hat{\mathbf{U}}$  denote the noisy state and joint distribution where the noise is  $\mathbf{M} = \mathbf{M}_{\lambda ,\mathbf{b}}$  , we have
$$
\begin{array}{l} \operatorname {S u r p} ^ {*} (\hat {\mathbf {w}}, \hat {\mathbf {U}}) = \det  (\hat {\mathbf {U}}) ^ {- \frac {1}{2 (| \Sigma | - 1)}} (\hat {\mathbf {w}} - \hat {\mathbf {q}}) \\ = \det  \left(\mathbf {M} ^ {\top} \mathbf {U M}\right) ^ {- \frac {1}{2 (| \Sigma | - 1)}} (\hat {\mathbf {w}} - \hat {\mathbf {q}}) \\ = \det  (\mathbf {U}) ^ {- \frac {1}{2 (| \Sigma | - 1)}} \det  (\mathbf {M}) ^ {- \frac {1}{| \Sigma | - 1}} (\hat {\mathbf {w}} - \hat {\mathbf {q}}) \\ = \frac {1}{1 - \lambda} \det  (\mathbf {U}) ^ {- \frac {1}{2 (| \Sigma | - 1)}} (\hat {\mathbf {w}} - \hat {\mathbf {q}}) (Claim6) \\ = \frac {1}{1 - \lambda} \det  (\mathbf {U}) ^ {- \frac {1}{2 (| \Sigma | - 1)}} (\mathbf {w} - \mathbf {q}) \mathbf {M} (Claim 5) \\ = \det  (\mathbf {U}) ^ {- \frac {1}{2 (| \Sigma | - 1)}} (\mathbf {w} - \mathbf {q}) \left(\mathbf {I} + \frac {\lambda}{1 - \lambda} \mathbf {B}\right) \\ = \det  (\mathbf {U}) ^ {- \frac {1}{2 (| \Sigma | - 1)}} (\mathbf {w} - \mathbf {q}) + \frac {\lambda}{1 - \lambda} \det  (\mathbf {U}) ^ {- \frac {1}{2 (| \Sigma | - 1)}} (\mathbf {w} - \mathbf {q}) \mathbf {B} \\ = \det  (\mathbf {U}) ^ {- \frac {1}{2 (| \Sigma | - 1)}} (\mathbf {w} - \mathbf {q}) \quad (\mathbf {w} \mathbf {B} = \mathbf {b} = \mathbf {q} \mathbf {B} \text {b y d e f i n i t i o n o f} \mathbf {B}) \\ = \operatorname {S u r p} ^ {*} (\mathbf {w}, \mathbf {U}). \\ \end{array}
$$
Thus, we finish the analysis for the invariant surprisal vector.
C.4 Proof of Theorem 2 and Theorem 3
In this section, we will show the omitted proof of Theorem 2 and Theorem 3. Before proving Theorem 2 and Theorem 3, we first analyse the results by assuming we know  $\hat{\mathbf{P}}$  in all scenarios.
Lemma 1 (Error probability, known  $\hat{\mathbf{P}}$ ). For any pair of papers  $A, B$  whose clean states are  $\mathbf{w}^A, \mathbf{w}^B$  correspondingly (without loss of generality, let  $\mathbb{E}_{\mathbf{w}^A}\varphi < \mathbb{E}_{\mathbf{w}^B}\varphi$ ), given their noise matrix  $\mathbf{M}^A, \mathbf{M}^B \in \mathcal{M}^*$  correspondingly,
$$
\operatorname * {P r} \left[ \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {A}} \varphi - \mathbb {E} _ {\hat {\mathbf {q}} ^ {A}} \varphi}{\operatorname * {d e t} (\hat {\mathbf {U}} ^ {A}) ^ {\frac {1}{2 (| \Sigma | - 1)}}} \geq \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {B}} \varphi - \mathbb {E} _ {\hat {\mathbf {q}} ^ {B}} \varphi}{\operatorname * {d e t} (\hat {\mathbf {U}} ^ {B}) ^ {\frac {1}{2 (| \Sigma | - 1)}}} \left| \mathbf {w} ^ {A}, \mathbf {w} ^ {B} \right] \right] \leq \exp \left\{\frac {- 2 \left(\mathbb {E} _ {\mathbf {w} ^ {B}} \varphi - \mathbb {E} _ {\mathbf {w} ^ {A}} \varphi\right) ^ {2} \operatorname * {d e t} (\mathbf {U}) ^ {- \frac {1}{| \Sigma | - 1}}}{\left(\varphi_ {\max } - \varphi_ {\min }\right) ^ {2} \left(\frac {1}{n ^ {A} (1 - \lambda^ {A}) ^ {2}} + \frac {1}{n ^ {B} (1 - \lambda^ {B}) ^ {2}}\right)} \right\}
$$
Proof of Lemma 1. Recall that  $\hat{v}$  is the empirical frequency of distribution  $\hat{w}$ . When we have infinite number of reviewers,  $\mathbb{E}_{\hat{\mathbf{v}}^A} \varphi = \mathbb{E}_{\hat{\mathbf{w}}^A} \varphi$  and  $\mathbb{E}_{\hat{\mathbf{v}}^B} \varphi = \mathbb{E}_{\hat{\mathbf{w}}^B} \varphi$ . Thus,
$$
\begin{array}{l} \Pr \left[ \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {A}} \varphi - \mathbb {E} _ {\hat {\mathbf {q}} ^ {A}} \varphi}{\operatorname* {d e t} (\hat {\mathbf {U}} ^ {A}) ^ {\frac {1}{2 (| \Sigma | - 1)}}} \geq \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {B}} \varphi - \mathbb {E} _ {\hat {\mathbf {q}} ^ {B}} \varphi}{\operatorname* {d e t} (\hat {\mathbf {U}} ^ {B}) ^ {\frac {1}{2 (| \Sigma | - 1)}}} \left| \mathbf {w} ^ {A}, \mathbf {w} ^ {B} \right. \right] \\ = \Pr \left[ \frac {\mathbb {E} _ {\hat {\mathbf {w}} ^ {A}} \varphi - \mathbb {E} _ {\hat {\mathbf {q}} ^ {A}} \varphi}{\det  (\hat {\mathbf {U}} ^ {A}) ^ {\frac {1}{2 (\left| \Sigma \right| - 1)}}} \geq \frac {\mathbb {E} _ {\hat {\mathbf {w}} ^ {B}} \varphi - \mathbb {E} _ {\hat {\mathbf {q}} ^ {B}} \varphi}{\det  (\hat {\mathbf {U}} ^ {B}) ^ {\frac {1}{2 (\left| \Sigma \right| - 1)}}} \left| \mathbf {w} ^ {A}, \mathbf {w} ^ {B} \right. \right] \\ = \Pr \left[ \mathbb {E} _ {\mathbf {w} ^ {B}} \varphi - \mathbb {E} _ {\mathbf {w} ^ {A}} \varphi \geq 0 \right] \quad (\text {C o r o l l a r y 1 : i n v a r i a n c e}) \\ = 1 \\ \end{array}
$$
However, when the number of reviewers is finite, we need to analyse the difference between  $\hat{w}$  and  $\hat{v}$  by concentration bound.
$$
\Pr \left[ \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {A}} \varphi - \mathbb {E} _ {\hat {\mathbf {q}} ^ {A}} \varphi}{\det  (\hat {\mathbf {U}} ^ {A}) ^ {\frac {1}{2 (| \Sigma | - 1)}}} \geq \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {B}} \varphi - \mathbb {E} _ {\hat {\mathbf {q}} ^ {B}} \varphi}{\det  (\hat {\mathbf {U}} ^ {B}) ^ {\frac {1}{2 (| \Sigma | - 1)}}} \left| \mathbf {w} ^ {A}, \mathbf {w} ^ {B} \right] \right.
$$
$$
= \Pr \left[ \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {A}} \varphi}{\det (\hat {\mathbf {U}} ^ {A}) ^ {\frac {1}{2 (| \Sigma | - 1)}}} - \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {B}} \varphi}{\det (\hat {\mathbf {U}} ^ {B}) ^ {\frac {1}{2 (| \Sigma | - 1)}}} \geq \frac {\mathbb {E} _ {\hat {\mathbf {q}} ^ {A}} \varphi}{\det (\hat {\mathbf {U}} ^ {A}) ^ {\frac {1}{2 (| \Sigma | - 1)}}} - \frac {\mathbb {E} _ {\hat {\mathbf {q}} ^ {B}} \varphi}{\det (\hat {\mathbf {U}} ^ {B}) ^ {\frac {1}{2 (| \Sigma | - 1)}}} \Big | \mathbf {w} ^ {A}, \mathbf {w} ^ {B} \right]
$$
$$
= \Pr \left[ \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {A}} \varphi - \mathbb {E} _ {\hat {\mathbf {w}} ^ {A}} \varphi}{\det  (\hat {\mathbf {U}} ^ {A}) ^ {\frac {1}{2 (| \Sigma | - 1)}}} - \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {B}} \varphi - \mathbb {E} _ {\hat {\mathbf {w}} ^ {B}} \varphi}{\det  (\hat {\mathbf {U}} ^ {B}) ^ {\frac {1}{2 (| \Sigma | - 1)}}} \geq - \frac {\mathbb {E} _ {\hat {\mathbf {w}} ^ {A}} \varphi - \mathbb {E} _ {\hat {\mathbf {q}} ^ {A}} \varphi}{\det  (\hat {\mathbf {U}} ^ {A}) ^ {\frac {1}{2 (| \Sigma | - 1)}}} + \frac {\mathbb {E} _ {\hat {\mathbf {w}} ^ {B}} \varphi - \mathbb {E} _ {\hat {\mathbf {q}} ^ {B}} \varphi}{\det  (\hat {\mathbf {U}} ^ {B}) ^ {\frac {1}{2 (| \Sigma | - 1)}}} \left| \mathbf {w} ^ {A}, \mathbf {w} ^ {B} \right] \right]
$$
$$
= \Pr \left[ \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {A}} \varphi - \mathbb {E} _ {\hat {\mathbf {w}} ^ {A}} \varphi}{\det  (\hat {\mathbf {U}} ^ {A}) ^ {\frac {1}{2 (| \Sigma | - 1)}}} - \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {B}} \varphi - \mathbb {E} _ {\hat {\mathbf {w}} ^ {B}} \varphi}{\det  (\hat {\mathbf {U}} ^ {B}) ^ {\frac {1}{2 (| \Sigma | - 1)}}} \geq - \frac {\mathbb {E} _ {\mathbf {w} ^ {A}} \varphi - \mathbb {E} _ {\mathbf {q}} \varphi}{\det  (\mathbf {U}) ^ {\frac {1}{2 (| \Sigma | - 1)}}} + \frac {\mathbb {E} _ {\mathbf {w} ^ {B}} \varphi - \mathbb {E} _ {\mathbf {q}} \varphi}{\det  (\mathbf {U}) ^ {\frac {1}{2 (| \Sigma | - 1)}}} \Big | \mathbf {w} ^ {A}, \mathbf {w} ^ {B} \right]
$$
(Corollary 1: invariance of our score)
$$
= \Pr \left[ \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {A}} \varphi - \mathbb {E} _ {\hat {\mathbf {w}} ^ {A}} \varphi}{\det  (\hat {\mathbf {U}} ^ {A}) ^ {\frac {1}{2 (| \Sigma | - 1)}}} - \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {B}} \varphi - \mathbb {E} _ {\hat {\mathbf {w}} ^ {B}} \varphi}{\det  (\hat {\mathbf {U}} ^ {B}) ^ {\frac {1}{2 (| \Sigma | - 1)}}} \geq \frac {\mathbb {E} _ {\mathbf {w} ^ {B}} \varphi - \mathbb {E} _ {\mathbf {w} ^ {A}} \varphi}{\det  (\mathbf {U}) ^ {\frac {1}{2 (| \Sigma | - 1)}}} \Big | \mathbf {w} ^ {A}, \mathbf {w} ^ {B} \right]
$$
$$
= \Pr \left[ \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {A}} \varphi - \mathbb {E} _ {\hat {\mathbf {w}} ^ {A}} \varphi}{\operatorname* {d e t} (\mathbf {M} ^ {A}) ^ {\frac {1}{| \Sigma | - 1}}} - \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {B}} \varphi - \mathbb {E} _ {\hat {\mathbf {w}} ^ {B}} \varphi}{\operatorname* {d e t} (\mathbf {M} ^ {B}) ^ {\frac {1}{| \Sigma | - 1}}} \geq \mathbb {E} _ {\mathbf {w} ^ {B}} \varphi - \mathbb {E} _ {\mathbf {w} ^ {A}} \varphi \Big | \mathbf {w} ^ {A}, \mathbf {w} ^ {B} \right] \quad (C l a i m 5: \hat {\mathbf {U}} = \mathbf {M} ^ {\top} \mathbf {U M})
$$
$$
= \Pr \left[ \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {A}} \varphi - \mathbb {E} _ {\hat {\mathbf {w}} ^ {A}} \varphi}{1 - \lambda^ {A}} - \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {B}} \varphi - \mathbb {E} _ {\hat {\mathbf {w}} ^ {B}} \varphi}{1 - \lambda^ {B}} \geq \mathbb {E} _ {\mathbf {w} ^ {B}} \varphi - \mathbb {E} _ {\mathbf {w} ^ {A}} \varphi \Big | \mathbf {w} ^ {A}, \mathbf {w} ^ {B} \right]
$$
$$
(\text {C l a i m} 6: \det  (\mathbf {M} _ {\lambda , \mathbf {b}}) = (1 - \lambda) ^ {| \Sigma | - 1})
$$
$$
= \Pr \left[ \left(\frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {A}} \varphi}{1 - \lambda^ {A}} - \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {B}} \varphi}{1 - \lambda^ {B}}\right) - \mathbb {E} \left[ \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {A}} \varphi}{1 - \lambda^ {A}} - \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {B}} \varphi}{1 - \lambda^ {B}} \right] \geq \mathbb {E} _ {\mathbf {w} ^ {B}} \varphi - \mathbb {E} _ {\mathbf {w} ^ {A}} \varphi | \mathbf {w} ^ {A}, \mathbf {w} ^ {B} \right]
$$
$(\mathbb{E}_{\mathbf{w}}\varphi = \sum_{s}\varphi (s)w_{s}$  and  $\hat{v}$  is the empirical frequency of distribution  $\hat{w})$
We expand  $\hat{v}^A$  and  $\hat{v}^B$  as the sum of independent random variables.
$$
\begin{array}{l} \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {A}} \varphi}{1 - \lambda^ {A}} - \frac {\mathbb {E} _ {\hat {\mathbf {v}} ^ {B}} \varphi}{1 - \lambda^ {B}} \\ = \frac {\sum_ {i = 1} ^ {n ^ {A}} \varphi \left(x _ {i} ^ {A}\right)}{n ^ {A} \left(1 - \lambda^ {A}\right)} - \frac {\sum_ {j = 1} ^ {n ^ {B}} \varphi \left(x _ {j} ^ {B}\right)}{n ^ {B} \left(1 - \lambda^ {B}\right)} \\ = \sum_ {i = 1} ^ {n ^ {A}} \frac {\varphi \left(x _ {i} ^ {A}\right)}{n ^ {A} \left(1 - \lambda^ {A}\right)} - \sum_ {j = 1} ^ {n ^ {B}} \frac {\varphi \left(x _ {j} ^ {B}\right)}{n ^ {B} \left(1 - \lambda^ {B}\right)} \\ \end{array}
$$