SlowGuess/NeurIPS_2023_OCR huggingface dataset:
"Cheap Signals" in Peer Review without a Prior
Yuxuan Lu
Center on Frontiers of Computing Studies
yx_lu@pku.edu.cn
Yuqing Kong*
yuqing.kong@pku.edu.cn
Peer review lies at the core of the academic process, but even well-intentioned reviewers can still provide noisy ratings. While ranking papers by average ratings may reduce noise, varying noise levels and systematic biases stemming from "cheap" signals (e.g. author identity, proof length) can lead to unfairness. Detecting and correcting bias is challenging, as ratings are subjective and unverifiable. Unlike previous works relying on prior knowledge or historical data, we propose a one-shot noise calibration process without any prior information. We ask reviewers to predict others' scores and use these predictions for calibration. Assuming reviewers adjust their predictions according to the noise, we demonstrate that the calibrated score results in a more robust ranking compared to average ratings, even with varying noise levels and biases. In detail, we show that the error probability of the calibrated score approaches zero as the number of reviewers increases and is significantly lower compared to average ratings when the number of reviewers is small.
Peer review is fundamental to the scientific process. However, as noted in Rennie [1]:
But it is a human system. Everybody involved brings prejudices, misunderstandings, and gaps in knowledge, so no one should be surprised that peer review is often biased and inefficient ... even with the best of intentions ...
Let us consider the following scenario.
Example 1 (Peer review). We have two papers and each paper is reviewed by 5 reviewers. We can only accept one paper. The first paper receives 4 "accept" and 1 "reject", the second paper receives 3 "accept" and 2 "reject".
A typical choice is to accept the first paper since it receives more fraction of "accept". Ideally, we should accept the paper which receives more fraction of "accept" in expectation from reviewers who have full expertise and invest full efforts, called the clean setting. However, in practice, the ratings can be noisy due to the lack of expertise or effort. If the noise is consistent for the two papers, the better paper should still receive more fraction of "accept" in expectation. Nevertheless, the noise can be different for different papers.
Example 2 (Hot vs. cold topic). The topic of the first paper is more popular than the second paper. In this case, the first paper is easier to obtain reviewers with high expertise. Thus, in the noisy setting, the first paper has less noisy ratings than the second paper.
It is possible that in the clean setting, the first paper receives  $80\%$  of "accept" in expectation and the second paper receives  $90\%$ . However, in the noisy setting, the first paper may still obtain  $80\%$  "accept" in expectation while the second paper may only obtain  $60\%$  "accept" in expectation because it receives more noisy ratings. In addition to the noise level, the noise can be systematically biased due to the existence of "cheap signals".
Example 3 (Long proof vs. short proof). The first paper has a complicated proof. The second paper has a short proof. In the noisy setting, each reviewer (unconsciously) intends to vote for "accept" for paper with a long proof, even if the length of the proof is a "cheap signal".
It may occur in practice that the second paper is better in the clean setting, but the first paper receives more fraction of "accept" in expectation in the noisy setting, due to the systematically biased noise caused by cheap signals. Cheap signals can also be the authors' reputation, institutions, or writing skills. One may argue that the cheap signals may correlate with expensive signals, which represent the true quality of the paper. However, these cheap signals like proof length can be easily manipulated.
Prior knowledge and historical data may help to calibrate the noise. However, the identification of bias can be subjective and ad hoc. It will be more applicable if we do not need prior knowledge and design a general one-shot process. A fundamental question arises:
*Can we design a one-shot scoring process that leads to a noise-robust rank without any prior knowledge? Formally, we want the paper with a higher expected score in the clean setting also have a higher score in the noisy setting with high probability, even if different papers' reviews have different noise levels and biases.
A key challenge is that the ratings are subjective and unverifiable. As an outsider without any prior knowledge, we do not even know whether the bias exists or not. Therefore, it is impossible to address the above question without eliciting any additional information from the reviewers. Prelec [2] propose an innovative method, Surprisingly Popular (SP), to aggregate the information despite agents' systematic biases without any prior. Here each agent is asked to report both her signal (e.g. accept or reject) and her prediction (e.g.  $90\%$  reject,  $10\%$  accept) for a random peer's report. The elicited predictions are used to construct the prior. Instead of a majority vote, the aggregation is the signal which is more popular compared to the prior<sup>2</sup>. By assuming that the ground truth is positively correlated to each agent's signal<sup>3</sup>, the aggregation is correct when the size of the crowds is sufficiently large.
We adopt the signal-prediction framework of SP, i.e., asking each reviewer to additionally provide her prediction for a randomly selected reviewer's report. Besides, we follow the literature of information elicitation [3] and information aggregation [4] to assume that agents are perfect Bayesian, which establishes a start for theoretical analysis:
Assumption 1 (Bias generated from "cheap signals" can be predicted). (Informal) We assume agents are perfect Bayesian. Besides, when their ratings are noisy and have systematic biases, they will adjust their predictions based on the biased noise.
The above assumption essentially provides a general definition of the bias. With this assumption, we aim to design a one-shot scoring process with the reviewers' predictions as calibrations.
We cannot directly apply SP to this scenario, not only because it requires a large number of agents, but also because it is designed to solve a problem with objective ground truth. In the peer review setting, each paper's ratings are subjective and do not need to be the same even in the clean setting. We may ask the reviewer to compare two papers and apply SP to aggregate the comparisons. However, in practice, papers are reviewed independently by a possibly different set of reviewers.
Moreover, it is possible that both two papers' surprisingly popular ratings are "accept". To rank them, one idea is to compare the "amount" of surprise. For example, SP calibrates the ratings by dividing the prior, where the prior is constructed by the predictions. Subtracting the prior is another option. However, neither of the two calibration options is robust to the noise. In Example 2, when the prior is  $50\% / 50\%$  for both the clean setting and the noisy setting, both of the two papers' surprisingly popular ratings are "accept". However, the second paper's "accept" is less popular compared to the prior because its ratings have more noise, while the second paper is better in the clean setting.
2 In a  $10\%$  reject and  $90\%$  accept prior,  $15\%$  reject and  $85\%$  accept ratings make "reject" surprisingly popular.  $\mathrm{Pr}[G = a|Y = a] > \mathrm{Pr}[G = a|Y = r]$ , where  $Y$  is a random agent's signal and  $G$  denotes the ground truth.
We formulate the above problem into a formal mathematical model, defining a paper's true quality as its expected rating in the clean setting without noise, and assuming that the noise is non-degenerate and each reviewer's received noisy signal positively correlates with her clean signal<sup>4</sup>.
Our Results Within the model and assumptions, we propose a one-shot scoring process. When the number of reviewers is infinite, a paper with a higher score in the noisy setting will also have a higher true quality even if the papers have different biased noises. When the number of reviewers is finite, we provide a bound for the error probability and numerically show that our method beats the baseline, the average rating, in various situations. We also extend the results to non-binary setting naturally, with proper restrictions on the noise structures. Of an independent interest, we also provide surprisal measures that are invariant to noises.
High-level Ideas The baseline score is the average rating, e.g., the fraction of "accept" votes. In this paper, we define the Surprisal-based Score in the form of  $\frac{\text{baseline score} - \text{prior expected score}}{\text{correlation between ratings}}$ , where the correlation is measured by a power of the determinant of the joint distribution matrix between two reviewers' ratings. Intuitively, the subtraction of the prior will cancel the effect of the systematic bias, and the division of the correlation will compensate for varying noise levels across contexts. For example, when faced with noisy reviewers providing weak signals beyond the prior, the average rating and the prior will closely align, resulting in a small correlation between ratings. By normalizing using this small correlation, the surprisal score is scaled up to compensate for the strong noise.
Three Reviewers, Binary Ratings To illustrate our scoring process, we provide an example for the case of three reviewers, binary ratings.
Example 4 (Three reviewers, binary ratings). In the setting of binary ratings, the signal of reviewers is either "accept" (1) or "reject" (0). We call the reviewer who votes for "accept" as positive reviewer and the reviewer who votes for "reject" as negative reviewer. We map 3 "accept" signals to the highest score, and 3 "reject" signals to the lowest score. When there are at least an "accept" and a "reject", we construct a matrix  $\hat{\mathbf{P}}$ , where  $\hat{P}_{s,t}$  is the average prediction, from the reviewers who report signal  $s$ , for the probability that a random reviewer votes for signal  $t$ . For example,  $\hat{P}_{0,1}$  is the average of the negative reviewers' predictions for the probability that a random reviewer votes for "accept".
Specifically, the score is defined as
$$
\mathrm {S} (\cdot) = \left\{ \begin{array}{l l} + \infty & 3 “ a c c e p t ”, 0 “ r e j e c t ” \\ (\frac {2}{3} - \hat {q} _ {1}) \big (\hat {q} _ {1} \hat {q} _ {0} \big (\hat {P} _ {1, 1} - \hat {P} _ {0, 1} \big) \big) ^ {- \frac {1}{2}} & 2 “ a c c e p t ”, 1 “ r e j e c t ” \\ (\frac {1}{3} - \hat {q} _ {1}) \big (\hat {q} _ {1} \hat {q} _ {0} \big (\hat {P} _ {1, 1} - \hat {P} _ {0, 1} \big) \big) ^ {- \frac {1}{2}} & 1 “ a c c e p t ”, 2 “ r e j e c t ” \\ - \infty & 0 “ a c c e p t ”, 3 “ r e j e c t ” \end{array} \right.
$$
where  $\hat{q}_1 = \frac{\hat{P}_{1,1}}{\hat{P}_{0,1} + \hat{P}_{1,0}}$  and  $\hat{q}_0 = \frac{\hat{P}_{1,0}}{\hat{P}_{0,1} + \hat{P}_{1,0}}$ .
Figure 1 illustrates how the score depends on the reviewers' predictions. At a high level, the score will be higher when the reviewers underestimate the probability that a random reviewer rates "accept" (the lower left corner)<sup>5</sup>.
To the best of our knowledge, we are the pioneer to rigorously formulate the noise-robust comparison problem with a one-shot scoring process, when different rating tasks have different noises. We believe our work provides a start in the peer review setting for calibrating the bias without any prior.
Bias in Peer Review Studies on peer review originated with Mahoney [5] and Hess [6]. Since then, numerous studies have emerged to identify sources and impacts of bias in peer review [7, 8, 9, 10, 11, 12, 13]. In addition to exploring sources and impacts, recent studies
<sup>4</sup>If not, the noisy reviewer can flip the rating sometimes. In this case, it is impossible to design a noise-robust scoring process without prior knowledge.
In our model, in predicting the probability that a random reviewer votes for "accept", the positive reviewer will have a higher prediction than the negative reviewer. In practice, if this is not true, we will adopt the baseline score to rank the papers.
Figure 1: Contours of  $S$  (ratings, predictions): We fix ratings (left: 1 accept, 2 reject; right: 1 reject, 2 accept) and illustrate the relationship between the score and the reviewers' predictions for the probability that a random reviewer votes for "accept".
have attempted to detect or calibrate biases automatically, employing various methods such as optimization, nonparametric estimation, and mechanism design [14, 15, 16, 17, 18, 19, 20]. However, these methods rely on historical data or a reviewer's votes for multiple papers.
Mechanism Design via Second-order Information Prelec [3] introduced the signal-prediction framework where agents are asked to provide both their answers and predictions. By applying the signal-prediction framework, researchers have discovered methods to aggregate people's answers to the surprisingly popular option [2], reduce bias by agents' predictions about ranking [21], incentivise truthful agents[3, 22, 23], and find more informative answers [24, 25]. In addition to the discrete setting, recently, a growing literature [26, 27, 28, 29, 30, 31] has extended the signal-prediction framework to forecast aggregation settings. In this work, we employ second-order information for noise-robust comparison in a different setting.
2 Background and Problem Statement
2.1 Background: Single-task Information Elicitation
In this section, we describe how agents receive private signals and generate predictions, following the standard signal-prediction framework [2, 3].
Model We regard each paper as a task with a state  $\mathbf{w} \in [0,1]^{1 \times |\Sigma|}$ , where  $\Sigma$  is the set of all possible signals. There are  $n$  agents (reviewers) assigned to the task $^6$ , labeled from 1 to  $n$ . Each agent  $i \in [n]$  receives private signal  $X_i = s$  with probability  $w_s$ . The notation  $x_i$  is the realization of  $X_i$ , representing the signal actually received by agent  $i$ . In this paper, we use the notation  $s$  or  $t$  to refer to a particular signal and the notation  $i$  or  $j$  to refer to a particular agent. There is a prior distribution  $Q$  over the states and  $W$  is a random state drawn from the prior distribution  $Q$ . We use a row vector  $\mathbf{q} \in [0,1]^{1 \times |\Sigma|}$  to denote the prior probability that an agent will receive each signal, i.e., for all  $s \in \Sigma$ ,  $q_s = \operatorname{Pr}_Q[X_i = s] = \sum_{\mathbf{w}} \operatorname{Pr}_Q[W = \mathbf{w}] w_s$ . Analogously,  $\mathbf{P} \in [0,1]^{|\Sigma| \times |\Sigma|}$  denotes the prediction matrix such that  $P_{s,t} = \operatorname{Pr}_Q[X_j = t | X_i = s], i \neq j$  is the probability that agent  $j$  will receive the signal  $t$  conditioning on another agent  $i$  receives the signal  $s$ . The joint distribution matrix  $\mathbf{U} \in [0,1]^{|\Sigma| \times |\Sigma|}$  is the matrix where  $\forall s, t \in \Sigma, U_{s,t} = q_s P_{s,t} = \operatorname{Pr}_Q[X_i = s, X_j = t]^7$ . Because agents' identities do not matter, we omit  $i, j \in [n]$  in the definition. To avoid degeneration, we assume the determinant of  $\mathbf{U}$  is non-zero. In the task, each agent  $i$  is asked to report her signal  $s_i$  (e.g. accept) and her prediction  $\mathbf{p}_i$  for a random agent's signal (e.g.  $40\%$  accept,  $60\%$  reject) without any communication to other agents.
The above model describes how agents receive signals without noise, which we call the clean setting. Figure 2 illustrates the model briefly and we offer a detailed example in Appendix A. We will introduce the noise model later.
The number of reviewers can be flexible for different papers.
The matrix  $\mathbf{U}$  is always symmetric but  $\mathbf{P}$  may not be symmetric.
Figure 2: Model without noise:  $Q$  is the prior distribution over the papers' states. In the figure, there are three types of states, each of which shows up with probability  $\frac{1}{3}$ . A random paper's state follows the distribution  $Q$ . When the paper's state realizes as  $\mathbf{w} = [0.2,0.8]$ , a random reviewer of this paper will vote for "reject" with probability 0.2 and vote for "accept" with probability 0.8.
Model Inference For each agent  $i \in [n]$ , given her private signal  $s_i$ , we assume that her prediction  $\mathbf{p}_i$  is the Bayesian posterior, i.e.,  $\forall t \in \Sigma, \mathbf{p}_i(t) = P_{s_i,t}$ . If for all signal  $s \in \Sigma$ , there exists an agent who reports  $s$ , then we can obtain the prediction matrix  $\mathbf{P}$  according to the agents' predictions. We can use  $\mathbf{P}$  to induce the joint distribution matrix  $\mathbf{U}$  where  $\forall s, t \in \Sigma, U_{s,t} = q_s P_{s,t} = \operatorname*{Pr}_Q[X_i = s, X_j = t]$ .
Claim 1 (Prelec et al. [2]). We can construct the vector  $\mathbf{q}$  and the joint distribution matrix  $\mathbf{U}$  from the prediction matrix  $\mathbf{P}$ :  $\forall s, t \in \Sigma, q_s = \left( \sum_{t} \frac{P_{s,t}}{P_{t,s}} \right)^{-1} \left( \frac{0}{0} \equiv 0 \right), U_{s,t} = q_s P_{s,t}$ .
In this paper, we will assume the reviewers have good intentions and will tell the truth. That is, if she receives an "accept" signal for a paper, she will vote for "accept" as well. Regarding incentive design, prediction reports can be incentivized by proper scoring rules [32]. For the signal report, Prelec [3] develops a payment method called Bayesian Truth Serum (BTS) to incentivize agents to tell the truth when the number of agents is infinite. By assuming that agents perform the same strategy, Kong et al. [33] propose a method that only requires the number of agents greater or equal 6.
2.2 Problem Statement
Noise Model In the noisy setting, instead of observing the original signal  $s$ , which we call the clean signal, each agent can only observe its noisy version  $M(s)$ , which we call the noisy signal, where  $M: \Sigma \mapsto \Sigma$  is a random operator. Denote  $\mathbf{M}$  as the matrix where  $M_{s,t}$  is the probability of observing noisy signal  $t$  given clean signal  $s$ . We only consider the non-degenerate noise whose noise matrix  $\mathbf{M}$  is invertible. Additionally, we focus on the homogeneous noise which corresponds to systematic biases. Homogeneous noise means that the noise is the same for all reviewers of a specific paper.
In the noisy setting, we use the notation  $\hat{Q}$  to denote the prior distribution over the noisy states. Similar to the clean setting,  $\hat{X}_i$  denotes the random variable of agent  $i$ 's noisy signal and  $\hat{x}_i$  is its realization;  $\hat{\mathbf{U}}$  denotes the joint distribution matrix where  $\hat{U}_{s,t} = \operatorname*{Pr}_{\hat{Q}}[\hat{X}_i = s,\hat{X}_j = t]$ ;  $\hat{\mathbf{w}}$  denotes the noisy state;  $\hat{\mathbf{q}}$  denotes the noisy prior over signals and  $\hat{\mathbf{P}}$  denotes the noisy prediction matrix. Note that the relationship between  $\hat{\mathbf{P}}$ ,  $\hat{\mathbf{U}}$  and  $\hat{\mathbf{q}}$  remains the same. By abusing notation a little bit, we sometimes write  $\hat{Q} = M(Q),\hat{\mathbf{U}} = M(\mathbf{U}),\hat{\mathbf{w}} = M(\mathbf{w})$ . Figure 3 briefly introduces how noise affects reviewers' ratings, and we offer a detailed example of noise model in Appendix A.
Assumption 2. In the noisy setting, we assume that for each agent  $i$ , given her private noisy signal  $s_i$ , her prediction  $\hat{\mathbf{p}}_i$  is still the Bayesian posterior where  $\forall t \in \Sigma, \hat{\mathbf{p}}_i(t) = \operatorname*{Pr}_{\hat{Q}}[X_j = t | X_i = s_i] = \hat{P}_{s_i,t}$ .
Ideally, we want to reconstruct the original state  $\mathbf{w}$  from the noisy  $\hat{\mathbf{w}}$  and  $\hat{\mathbf{P}}$ . However, we do not have a sufficient amount of information. Instead, we aim to solve the following problem.
Noise-Robust Comparison We have two rating tasks, say reviews for paper A and B, whose clean states are  $\mathbf{w}_A$  and  $\mathbf{w}_B$  respectively. Both  $\mathbf{w}_A$  and  $\mathbf{w}_B$  follow the distribution  $Q$ . There is a known one-to-one mapping  $\varphi: \Sigma \to \mathbb{R}$  between signals in  $\Sigma$  and paper scores. For example, when ratings are binary, the mapping  $\varphi$  is  $\{\mathrm{reject} \to 0, \mathrm{accept} \to 1\}$ .
Figure 3: Model with noise
Therefore, for simplicity, we sometimes write signal  $s$  as a score of real number and  $\Sigma$  as a set of scores. For example, in the binary case,  $\Sigma = \{0(\text{reject}), 1(\text{accept})\}$ . We care about the true quality which is defined as the expected score in the clean state, i.e.,  $\mathbb{E}_{\mathbf{w}}\varphi = \sum_{s}\varphi(s)w_{s}$ . For example, in the binary case, the quality is  $\mathbb{E}_{\mathbf{w}}\varphi = w_{1}$ , i.e., the probability of a random reviewer voting for "accept".
Paper A has  $n_A$  reviewers and paper B has  $n_B$  reviewers. They do not need to have the same number of reviewers. The pair  $(\hat{x}_i^A, \hat{\mathbf{p}}_i^A)_{i \in [n_A]}$  denotes the reviewers' actual ratings and predictions for paper A, and  $(\hat{x}_j^B, \hat{\mathbf{p}}_j^B)_{j \in [n_B]}$  analogously. Let  $S(\cdot)$  be a function that takes reviewers' noisy ratings and predictions of a single paper as input, and outputs a calibrated score for the paper. For a pair of papers, we will rank the paper according to their calibrated scores<sup>8</sup>. Given a set of noises  $\mathcal{M}$ , we aim to design  $S(\cdot)$  such that for all  $M_A, M_B \in \mathcal{M}$ , the error probability  $\operatorname*{Pr}[S(\cdot) \text{ ranks A higher than B}]$  is better than A's] is upper-bounded and the upper bound goes to 0 when both  $n_A$  and  $n_B$  go to infinity.
3 Invariant Surprisal Vector
This section will introduce the key ingredient of our scoring process, a surprisal vector which is invariant to a natural family of noises. We first introduce this family of noises. Here each individual receives the clean signal with probability  $1 - \lambda > 0$ . With probability  $\lambda$ , which we call noise level, each individual receives a signal according to a distribution vector  $\mathbf{b} \in \Delta_{\Sigma}$ , which we call bias. We state the noise formally in the following definition.
Definition 1 (A Family of Noises  $\mathcal{M}^*$ ). We consider a family of noises with two parameters, a noise level  $\lambda \in [0,1)$  and a bias vector  $\mathbf{b} \in \Delta_{\Sigma}$ , where  $\Delta_{\Sigma}$  is the probability simplex on the signal set  $\Sigma$ . Let  $\mathbf{B}$  denote the matrix whose rows are all  $\mathbf{b}$ . The noise is defined as  $\mathbf{M}_{\lambda,\mathbf{b}} = (1 - \lambda)\mathbf{I} + \lambda\mathbf{B}$ .
Claim 2. The noisy state  $\hat{\mathbf{w}} = (1 - \lambda)\mathbf{w} + \lambda \mathbf{b}$  is a convex combination of the clean state  $\mathbf{w}$  and the bias vector  $\mathbf{b}$ . In the binary case ( $\Sigma = \{0(\text{reject}), 1(\text{accept})\}$ ),  $\mathcal{M}^*$  is the set of all non-degenerate noises where  $M_{1,1} > M_{0,1}$ .
Then, we define a surprisal vector that is invariant to the above noises.
Definition 2 ( $\mathcal{M}^*$ -Invariant Surprisal Vector). Given paper's state  $\mathbf{w}$  and joint distribution  $\mathbf{U}$ , we define the  $\mathcal{M}^*$ -invariant surprisal vector as
$$
\operatorname {S u r p} ^ {*} (\mathbf {w}, \mathbf {U}) = \det  (\mathbf {U}) ^ {- \frac {1}{2 (| \Sigma | - 1)}} (\mathbf {w} - \mathbf {q}),
$$
where  $\mathbf{q} \in [0,1]^{1 \times |\Sigma|}$  denotes the marginal distribution of  $\mathbf{U}$ .
If two papers have the same score, we will randomly rank one higher, with a  $50\%$  chance for either.