Calibrating “Cheap Signals” in Peer Review without a Prior
Authors: Yuxuan Lu
Yuqing Kong
Venue: NeurIPS 2023 poster (2023)
Keywords: Peer prediction
Peer review
Calibration
TL;DR: When papers receive ratings of different noises due to heterogeneity of paper topics and the existence of cheap signals, we propose a scoring process that leads to a noise-robust rank among papers.

Abstract: Peer review lies at the core of the academic process, but even well-intentioned reviewers can still provide noisy ratings. While ranking papers by average ratings may reduce noise, varying noise levels and systematic biases stemming from ``cheap'' signals (e.g. author identity, proof length) can lead to unfairness. 
Detecting and correcting bias is challenging, as ratings are subjective and unverifiable. Unlike previous works relying on prior knowledge or historical data, we propose a one-shot noise calibration process without any prior information. We ask reviewers to predict others' scores and use these predictions for calibration. Assuming reviewers adjust their predictions according to the noise, we demonstrate that the calibrated score results in a more robust ranking compared to average ratings, even with varying noise levels and biases.
In detail, we show that the error probability of the calibrated score approaches zero as the number of reviewers increases and is significantly lower compared to average ratings when the number of reviewers is small.

Calibrating
"Cheap Signals" in Peer Review without a Prior
Yuxuan Lu Center on Frontiers of Computing Studies School of Computer Science
Yuqing Kong"
Center on Frontiers of Computing Studies
School of Computer Science
Peking University
Peking University
Beijing, China
Beijing, China
yx_ 1u@pku edu. cn
yuqing kong@pku edu. cn
Abstract
Peer review lies at the core of the academic process, but even well-intentioned reviewers still provide noisy ratings. While ranking papers by average ratings may reduce noise, varying noise levels and systematic biases stemming from "cheap" signals (e.g. author identity, proof length) can lead unfairness. Detecting and correcting bias is challenging, as ratings are subjective and unverifiable. Unlike previous works relying on prior knowledge or historical data, we propose sho-shot noise calibration process without any prior information. We ask reviewers to predict others' scores and use these predictions for calibration. Assuming reviewers adjust their predictions according to the noise, we demonstrate that the calibrated score results more robust ranking compared average ratings, even with varying noise levels and biases. In detail, we show that the eтror probability of the calibrated score approaches zero as the number of reviewers increases and significantly lower compared to average ratings when the number of reviewers is small.
Introduction
Peer review is fundamental to the scientific process. However, as noted in Rennie M:
But it is human system. Everybody involved brings prejudices, misunderstandings, and gaps in knowledge, so no one should be surprised that peer review is often biased and inefficient even with the best of intentions...
Let us consider the following scenario.
Example (Peer review). We have two papers and each paper is reviewed by 5 reviewers. We can only accept one paper. The first paper receives "accept" and 1 "reject", , the second paper receives 3 "accept" and 2 "reject".
typical choice is to accept the first paper since receives more fraction of "accept". Ideally, we should accept the paper which receives more fraction of "accept" in expectation from reviewers who have full expertise and invest full efforts, called the clean setting. However, in practice, the ratings can be noisy due the lack of expertise effort. If the noise is consistent for the two papers, the better paper should still receive more fraction of "accept" in expectation. Nevertheless, the noise be different for different papers.
Example 2 (Hot vs. cold topic). The topic of the first paper is more popular than the second paper. In this case, the first paper is easier obtain reviewers with high expertise. Thus, in the noisy setting. the first paper has less noisy ratings than the second paper.
Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

It is possible that in the clean setting, the first paper receives 80% of "accept" in expectation and the second paper receives 90%. However, in the noisy setting, the first paper may still obtain 80% "accept" in expectation while the second paper may only obtain 60% "accept" in expectation because it receives more noisy ratings. In addition to the noise level, the noise can be systematically biased due to the existence of "cheap signals".
Example 3 (Long proof vs. short proof). The first paper has complicated proof. The second paper has short proof. In the noisy setting, each reviewer (unconsciously) intends to vote for "accept" for paper with long proof, even if the length of the proof is a "cheap signal".
may occur in practice that the second paper is better in the clean setting, but the first paper receives more fraction of "accept" in expectation in the noisy setting, due to the systematically biased noise caused by cheap signals. Cheap signals can also be the authors' reputation, institutions, or writing skills. One may argue that the cheap signals may correlate with expensive signals, which represent the true quality of the paper. However, these cheap signals like proof length can be easily manipulated.
Prior knowledge and historical data may help to calibrate the noise. However, the identification of bias can be subjective and ad hoc. It will be more applicable if we do not need prior knowledge and design a general one-shot process. A fundamental question arises:
* "an we design one-shot scoring process that leads noise-robust rank without any prior knowledge? Formally, we want the paper with higher expected score in the clean setting also have a higher score in the noisy setting with high probability, even if different papers' reviews have different noise levels and biases.
A key challenge is that the ratings are subjective and unverifiable. As outsider without any prior knowledge, we do not even know whether the bias exists or not. Therefore, it is impossible to address the above question without eliciting any additional information from the reviewers. Prelec [2] propose an innovative method, Surprisingly Popular (SP), to aggregate the information despite agents' system- atic biases without any prior. Here each agent is asked report both her signal (e.g. accept reject) and her prediction (e.g. 90% reject, 10% accept) for a random peer's report. The elicited predictions used to construct the prior. Instead of a majority vote, the aggregation is the signal which is more popular compared to the priou"] By assuming that the ground truth is positively correlated to each agent's 's signa the aggregation is correct when the size of the crowds is sufficiently large.
We adopt the signal-prediction framework of SP, i.e. asking each reviewer to additionally provide her prediction for randomly selected reviewer's 's report. Besides, follow the literature of information elicitation B and information aggregation 4 to assume that agents are perfect Bayesian, which establishes a start for theoretical analysis:
Assumption 1 (Bias generated from "cheap signals" can be predicted). (Informal) We assume agents are perfect Bayesian. Besides, when their ratings are noisy and have systematic biases, they will adjust their predictions based on the biased noise.
The above assumption essentially provides general definition of the bias. With this assumption, we aim to design a one-shot scoring process with the reviewers' predictions as calibrations.
We cannot directly apply SP to this scenario, not only because it requires a large number of agents, but also because it is designed to solve problem with objective ground truth. In the peer review setting, each paper's ' ratings are subjective and do not need to be the same even in the clean setting. We may ask the reviewer to compare two papers and apply SP to aggregate the comparisons. However, in practice, papers are reviewed independently by a possibly different set of reviewers.
Moreover, it is possible that both two papers' surprisingly popular ratings are "accept". To rank them, one idea is to compare the "amount" of surprise. For example, SP calibrates the ratings by dividing the prior, where the prior is constructed by the predictions. Subtracting the prior is another option. However, neither of the two calibration options is robust to the noise. In Example 2 when the prior is 50% / 50% for both the clean setting and the noisy setting, both of the two papers' surprisingly popular ratings are "accept". However, the second paper's 's "accept" is less popular compared to the prior because its ratings have more noise, while the second paper is better in the clean setting.
In a 10% reject and 90% accept prior, 15% reject and 85% accept ratings make "reject" surprisingly popular. "rr[G=d| a|Y =a]> Pr[G=a/Y =r], where Y is random agent's signal and G denotes the ground truth.
2

---

We formulate the above problem into a formal mathematical model, defining a paper's ss true quality as its expected rating in the clean setting without noise, and assuming that the noise non-degenerate and each reviewer's received noisy signal positively correlates with her clean signa 4
Our Results Within the model and assumptions, we propose a one-shot scoring process. When the number of reviewers is infinite, paper with higher score in the noisy setting will also have a higher true quality even if the papers have different biased noises. When the number of reviewers is finite, we provide a bound for the enror probability and numerically show that our method beats the baseline, the average rating, in various situations. We also extend the results to non-binary setting naturally, with proper restrictions the noise structures. Of an independent interest, also provide surprisal measures that are invariant to noises.
High-level Ideas The baseline score is the average rating, e.g., the fraction of "accept" votes. In this paper, we define the Surprisal-based Score the form of baseline score prior expected score where the correlation between ratings correlation measured by power of the determinant of the joint distribution matrix between two reviewers' ratings. Intuitively, the subtraction of the prior will cancel the effect of the systematic bias, and the division of the correlation will compensate for varying noise levels across contexts. For example, when faced with noisy reviewers providing weak signals beyond the prior. the average rating and the prior will closely align, resulting in a small correlation between ratings. By normalizing using this small correlation, the surprisal score is scaled up compensate for the strong noise.
Three Reviewers, Binary Ratings
To illustrate our scoring process, we provide an example for the case of three reviewers, binary ratings.
Example 4 (Three reviewers, binary ratings). In the setting of binary ratings, the signal of reviewers is either "accept" (1) or "reject" (0). We call the reviewer who votes for "accept" as positive reviewer and the reviewer who votes for "reject" highest score, and 3 "reject" signals to the lowest score. When there are at least an "accept" and a "reject", we construct a matrix P, where Ps,, is the average prediction, from the reviewers who report signal s, for the probability that random reviewer votes for signal t. For example, Po. is the average of the negative reviewers' predictions for the probability that a random reviewer votes for "accept". Specifically, the score is defined as
negative reviewer. We map 3 "accept" signals to the
+00
3 "accept", 0 "reject"
2 " "accept", 1 "reject"
s(-)
én Pa.
1 "accept", 2 "reject"
0 "accept", 3 "reject"
P.a and qo
A.o
where Φ1
Ps.
Po.1+
Figure I illlustrates how the depends on the reviewers' predictions. At high level, the score will be higher when the reviewers underestimate the probability that a random reviewer rates "accept" (the lower left corner
To the best of our knowledge, we are the pioneer to rigorously formulate the noise-robust comparison problem with a one-shot scoring process, when different rating tasks have different noises. We believe our work provides a start in the peer review setting for calibrating the bias without any prior.
1.1 Related Work
Bias in Peer Review Studies on peer review originated with Mahoney [5] and Hess [ Since then, numerous studies have emerged to identify sources and impacts of bias in peer review
In addition to exploring sources and impacts, recent studies
'If not, the noisy reviewer can flip the rating sometimes. In this case, it is impossible to design a nobse-robust scoring process without prior knowledge.
"I our model, in predicting the probability that a random reviewer votes for "accept", the positive reviewer will have a higher prediction than the negative reviewer. In practice. if this is not true, we will adopt the baseline score rank the papers.
3

---

1 accept and 2 reject
accept and reject
:0.9
0.0
0.7
& 0.8
0.5
Postivv welswers
0.5
0.4
0.4
0.3
0.3
pestee
16
0.1 02 03 0.4 0.5 0.6 0.7 0.8 0.9
0.1 0.2 0:3 0.4 0.5 De 0.7
predicton PuL
Negative
Pt.1
soadings.predcttions) We fix ratings (left: accept, 2 reject: right: 1 reject, 2 accept) and illustrate the relationship between the score and the reviewers' predictions for the probability that a random reviewer votes for "accept".
Figure 1: Contours
have attempted to detect or calibrate biases automatically, employing various methods such as optimization, nonparametric estimation, and mechanism design [6 19 20 However,
these methods rely on historical data or reviewer's votes for multiple papers.
Mechanism Design via Second-order Information Prelec introduced the signal-prediction
framework where agents are asked to provide both their answers and predictions. By applying the signal-prediction framework, researchers have discovered methods to aggregate people's answers to the surprisingly popular option (2). reduce bias by agents' predictions about ranking (21), incentivise
truthful agents 3 22 23), and find more informative answers 24 25 In addition to the discrete
setting, recently, a growing literature [26] 27 28 29 30 31 has extended the signal-prediction framework to forecast aggregation settings. In this work, employ second-order information for noise-robust comparison in a different setting.
2 Background and Problem Statement
2.1 Background: Single-task Information Elicitation
In this section, describe how agents receive private signals and generate predictions, following the standard -redictpro framework 2.3).
Model We regard each paper as a task with state w e [0,1] where is the set of all possible signals. There are n agents (reviewers) assigned to the task labeled from to n. Each agent ie [m] receives private signal X,= with probability w.- The notation æ; is the realization of X,, representing the signal actually received by agent i. In this paper, we use the notation 8 or to refer particular signal and the notation or refer particular agent. There
prior distribution Q over the states and W is a random state drawn from the prior distribution Q. We use a row vector qe [0,1] 1x Σl to denote the prior probability that an agent will receive each signal, i.e., for all se Σ, q. = P.=Prol X, =s] ProlWs =w]w.. Analogously. Pe [0,1] denotes the prediction matrix such that P.. Prq[ t|X; = s],i # is the probability that agent j will receive the signal conditioning on another agent receives the signal s. The joint distribution matrix Ue [0(0,1] [=0xi))))x is the matrix where Vs./c E2U,,.=-1 Pa.: Pro X; xxx,= Because agents' identities do not matter, we omit [ in the definition. avoid degeneration, we assume the determinant U non-zero. In the task, each agent is asked to report her signal s; (e.g. accept) and her prediction p. for random agent's 's signal (c.g. 40% accept, 60% reject) without any communication to other agents.
The above model describes how agents receive signals without noise, which we call the clean setting. Figure illustrates the model briefly and we offer a detailed example in Appendix A We will introduce the noise model later.
Thee number reviewers can flexible different papers. The matrix U is always symmetric but P may not be symmetric.

---

[0.2.088)
0.2 Rej M
0.8
Acc
Figure 2: Model without noise: Q is the prior distribution over the papers' states. In the figure, there are three types of states, each of which shows up with probability 3. A random paper's 's state follows the distribution Q. When the paper's 's state realizes as w [0.2,0... random reviewer of this paper will vote for "reject" with probability 0.2 and vote for "accept" with probability 0.8.
Model Inference For each agent ie [n], given her private signal 8,, we assume that her prediction P: is the Bayesian posterior, i.e., VE (t) Ps.,t- If for all signal s E E, there exists an agent who reports s, then we can obtain the prediction matrix P according to the agents' predictions. We can use P to induce the joint distribution matrix U where Vs,te,,U, q. Pr.:=PPo(,,ss,X
Claim (Prelec et al. 21). We can construct the vector and the joint distribution matrix U from the prediction matrix P: Vs,te (E,
(E. Put
(0)0=)))) u..
In this paper, will assume the reviewers have good intentions and will tell the truth. That is, if she receives an "accept" signal for a paper, she will vote for "accept" as well. Regarding incentive design, prediction reports can be incentivized by proper scoring rules 332 For the signal report, Prelec 3 develops a payment method called Bayesian Truth Serum (BTS) to incentivize agents to tell the truth when the number of agents is infinite. By assuming that agents perform the same strategy, Kong et al.
3
propose method that only requires the number of agents greater or equal 6.
2.2 Problem Statement
Noise Model In the noisy setting, instead of observing the original signal s, which we call the clean signal, each agent can only observe its noisy version Ms), which we call the noisy signal, where M:E+>> Σ is random operator. Denote M as the matrix where M,, is the probability of observing noisy signal given clean signal s. We only consider the non non-degenerate noise whose noise matrix M is invertible. Additionally, we focus the homogeneous noise which corresponds to systematic biases. Homogeneous noise means that the noise is the same for all reviewers of a specific paper.
In the noisy setting, we use the notation to denote the prior distribution over the noisy states. Similar to the clean setting, X, denotes the random variable of agent i's noisy signal and á is its realization; Ö denotes the joint distribution matrix where Ú.,5 = Pr ProlX; = 8, =t]; w denotes the noisy state; a denotes the noisy prior over signals and P denotes the noisy prediction matrix. Note that the relationship between P, Ö and 9 remains the same. By abusing notation little bit, we sometimes write Q= (Q). Ú M(U), w (w). Figure B introduces how noise affects reviewers' ratings, and we offer detailed example of noise model in Appendix [
Assumption 2. In the noisy setting, we assume that for each agent i, given her private noisy signal 8:] Pa..t
8G, her prediction Da is still the Bayesian posterior where VL
Ideally, we want to reconstruct the original state w from the noisy w and P. However, we do not have a sufficient amount of information. Instead, we aim to solve the following problem.
have two rating tasks, say reviews for paper A and B, whose clean states are w and w respectively. Both w and w follow the distribution Q. There is a known one-to-one mapping :p: Σ - between signals in E and paper scores. For example, when ratings are binary, the mapping yo is {reject- 0, accept
Moise-Robust Comparison
5

---

W [0.2.0.8]
[.6.0.4
[0.8002]
[0.4,0.6]
0.5
M
L0.5
0.5
0.6
0.4
0.4
0.6
Rej
M
Acc
Rej
11
Acc
Negative bias of reviewers' ratings of paper with state w= (0.200...8 there a 50% chance that reviewer will vote for "reject" without reading it carefully. In this case, random reviewer of this paper will vote for "accept" with probability 0.4.
(h) Positive bias of reviewers' ratings of paper with state w= [0.8,022]: there is 50% chance that re- viewer will vote for "accept" without reading it care- fully. In this case, a random reviewer of this paper will vote for "accept" with probability 0.6.
Figure 3: Model with noise
Therefore, for simplicity, we sometimes write signal s as a score of real number and Σ set of scores. For example, in the binary case, Σ= [=((oreject) (ecejjec),,(acceee)) (accept)). We care about the true quality which is defined as the expected score in the clean state, i.e., g= E. ( ()... For example, the binary case, the quality is =w i... the probability of random reviewer voting for "accept".
Paper A has nA reviewers and paper B has ng reviewers. They do not need to have the same number of reviewers. The pair (#: (e()))(a)) denotes the reviewers' actual ratings and predictions for paper A, and (&f p/ je[ng] analogously. Let S(.) be function that takes reviewers' noisy ratings and predictions of a single paper as input, and outputs a calibrated score for the paper. For pair of papers,
will rank the paper according to their calibrated scores Given set of noises M, we aim to design S(.) such that for all MA, MB e M,, the error probability Pr(S(.) ranks A higher than B quality is better than A's] is upper-bounded and the upper bound goes to 0 when both A and n. go to infinity.
Invariant Surprisal Vector
This section will introduce the key ingredient of our scoring process, a surprisal vector which is invariant to natural family of noises. We first introduce this family of noises. Here each individual receives the clean signal with probability 1-Àx With probability x, which we call noise level, distribution vector be which call bias. We
each individual receives signal according state the noise formally in the following definition.
Definition (A Family of Noises M*). We consider a family of noises with two parameters, a noise level λe (0,1) and bias vector be Ag, where Ag is the probability simplex on the signal set E. Let B denote the matrix whose rows are all b. The noise is defined as M (1-==(1+)I+AAB
Claim 2. The noisy state w (1 A)w + Xb is a convex combination of the clean state w and the bias vector b. In the binary case (E= (etrejeccr).((aeeeppee noises where M1,1 > M0.1-
M is the set of all non-degenerate
Then, we define surprisal vector that is invariant to the above noises.
Definition 2 (M* IIvaarann Surprisal Vector). Given paper's state w and joint distribution U, we define the -iniaaariant surprisal vector as
Surp'
where [0(001]1xP) denotes the marginal distribution of U.
"If' two papers have the same score, will randomly rank higher, with 50% chance for either.

---

Notation
Description
Q.Q
Q denotes the distribution over the clean states. Q denotes the distribution over the noisy states.
w,W
denotes the paper's clean state. denotes the paper's noisy state.
X, denotes a random variable indicating reviewer i's clean signal for a paper which state follows
X Ti,li
Q. X. denotes a random variable indicating reviewer i's noisy signal for a paper which state follows Q. Fi,5i their realizations respectively.
U denotes the prior joint distribution over two reviewers' ratings, where Ua,: = Pro(X,
U
8, s,X,=t]. Ö denotes its noisy version that P
denotes the prior distribution vector over a reviewer's vote, where 9, = Pro[ = s]. q denotes its noisy version that -.= [r.prr([,,==
q.9
denotes the prediction matrix, where Pa.e= = = [to][Xx==X;x=s]]]] P denotes its noisy
P.P
version where Pr.=Prol(,=ttXeee
n
denotes the number of reviewers that are assigned for paper.
M
M is the noise matrix defined by M= b, where is the noise level and b is the hias vector.
The frequency vector of the ratings, where v, = = - = s]. v denotes noisy version that
v,Y
h== [$.=s].
(*()
S*(+) is our Surprisal-basee based Score calculated from w and U.
S()
S(-) is our Empirical Surprisal based Score calculated from n agents' ratings and predictions. Table I: Notation table
Claim 3. In the binary case (E'= (0(rejectt, the M*-invarian surprisal vector can be simplified as
Surp" (w,U)=
w w-q q
4uuqi (Po, Po.1)
Theorem 1 (Invariance). The M -invariaan surprisal is non-negative, vanishes when the state w equals the prior q, and is invariant to My,b for all Xe [0,1), be The implication of invariance is that for any noise with (0,1) Surp" (w, (w,,U)Suurp Surp=""ur"">>a (w,
The proofs of Claim 2 Claim 3 and Theorem [
aare are deferred Appendix C For general noise, the direction of w-q may not be invariant. For example, one possible noise is flipping the ratings, i.c., rating "accept" when "reject" and rating "reject" when "accept". In this case, the direction of w-q will be flipped as well. However, if we only care about the amount of surprisal, we successfully design measure that invariant to all possible non-degenerate noise and show that can be used to identify the true state w in some scenarios (see Appendix B
4 Surprisal-based Score
This section introduces our scoring process based on the surprisal vector introduced in Section 3 We first introduce the score based on "wishful thinking", i.e., when we have the state (or the noisy state). We then resolve this "wishful thinking" by providing an estimation of the state.
Definition 3 (Surprisal-basee Score). Given and U, we define the M -invariant Surprisal-basee Score as
S*W,UU) (Ewip- det U) TI
According to the simplification in the binary case (Claim 3 the M -invariant surprisal score in the binary case can be simplified to
W1 91
S*(WUU)=
9091 (Pi,1 Po.1)
Theorem [ directly implies the following results: given the noisy states, we can compare their Surprisal-based Scores S* (w, U) and the comparison is invariant to the noise in M and consistent to the comparison of their true qualities.
7

---

Corollary 1 invarainv comparison by S²(-)), For any two states WA. WB which follow distribution Q, and any two noises M A
S* (MM,(Wa),MM(U)) (MM
(Ms(WE).MM(())
=S' -S' -S (wB.U <c 5 E
When there are only small number of reviewers, we will use an estimation of the Surprisal-based Score, which we call the Empirical Surprisal-based Score. We first focus on the binary case. We have already proved that for any invertible noise matrix M where M1.1 My.1, the metric can be used to implement noise-invariant comparison (Claim 2 Corollary II.
w1-M1
((gogi (F%,1-Po11)
We will use the reviewers' ratings and predictions to estimate the score. We employ a frequeney vector V E [0,1] | to denote the frequency of the reviewers' reported signals. Formally, for all SE 02- - Ex- [
s]. where é; is the reported noisy signal reviewer i. For example, in the binary case, v= 01) where 01 is the fraction of "accept" and 0g is the fraction of "reject".
Score (Binary) When there exists at least one "accept" and one "reject", we construct 2x2 prediction matrix P based on the reviewers' ratings and predictions. Each element the matrix, Ps.., is the average prediction from the reviewers who report the signal s, for the probability that a random reviewer reports the signal t. For example, Po. is the average of the negative reviewers' prediction for the probability a random reviewer votes for "accept". When reviewers' ratings are the same, the matrix P cannot be constructed. In this case, we set + +o score for the papers that all reviewers vote for "accept" and -oo for the papers that all reviewers vote for "reject". Formally,
Empirical Surprisal-
1 0
S( S((#;:D)))(m))
1 (0.1),
(in +000
===1
A. where @1 = Po
and qu=
Po, (See Claim for the calculation of from P).
A.o
We naturally extend the process to non-binary settings. For example, review signals contain multiple grades (E=e(2(reject), -wwakk reject) ,1 (weak accept), (acceptt)) ))
Empirical Surprisal-based Score (General) When ig >0 0 Vs E, we construct the prediction matrix P. The calibrated score is defined
S(( (BBp "
where Vse E.g. (E. (0/0= When de)(())+ 00, S(.) remains undefined. This "bad event" arises when a reviewer favors a paper, her belief in the likelihood of another reviewer also favoring decreases. This unusual scenario alerts the Program Committee that this particular paper warrants further discussion and careful consideration before making a final decision. When comparing tasks with undefined scores, our method degenerates to the baseline, i.e., comparing the value of Egy.
5 Theoretical Guarantee
In this section, we theoretically analyse the performance of our Empirical Surprisal- baased Score. We use the error probability to measure the performance. Recall that use the expected score of a paper in the clean state to measure its true quality. In the binary case, the true quality of a paper, whose clean state is w, is W1, i.e., the fraction of "accept" ratings in the clean setting. The error probability the probability that the score ranks paper A higher than paper B while 's quality is better than A's.
Theorem 2 (binary) and Theorem 3 (general) show the theoretical upper bound of the error probability of our method. When the number reviewers goes to infinity, the error probability goes to zero. The analysis follows from the results of Corollary [T and a standard concentration bound analysis. We defer the proofs of these two thcorems to Appendix
Theorem 2 (Error probability, binary case). For any pair of papers A,B whose clean states are w B correspondingly (without loss of generality, let w w ), given their noise matrices
8

---

M .M MM" correspondingly the error probability Pr[S(A) S(B) w w Pr(S(A)=
((B) _wf] goes O when the number of reviewers nLA,BB goes to infinity, and is bounded by
(1of) + (1-101) an - (11)) () (1-101 (1-101
2( TuP-w/ w1)²
exp
aa(1-Aa)² -
where XA is the noise level of paper A and Xg is the noise level of paper B
The error bound has three terms. The first two terms bound the error probability conditioning on at least one of the papers having an infinite score, i.e., every reviewer votes for "accept" ("reject"). The last term bounds the error probability conditioning on both papers has at least an "accept" and a "reject" vote. This theorem shows that can achieve better error bound when the rating difference (wf -wy") between two papers is larger, when there are more reviewers (NA and ng), and when the noise levels (A and λB) are lower.
Theorem 3 (Error probability, general case). For any pair of tasks A,B whose clean states are W w correspondingly (without loss of generality, let E. (), given their noise matrices MA, NA,MB= EM* correspondingly, the error probability Pr[S(A) > S(B)(wA .w Pr(S(AA)
S(B)lwA ww" goes to O when the number of reviewers A, goes infinity, and is bounded by
2(E ng n B
(1-wA)) M (1-60 )) +exp
(Pmax Painn)mmm))) na(1-Da)² "n((-AB))"
6 Numerical Experiments
We perform numerical experiments to compare the performance of our Surprisal-basee Score and the baseline score in the binary setting (-= (O(reject), ,I (accept))). Recall that the baseline is the pro- portion of the "accept" ratings. Here we describe the parameters we select in numerical experiments.
1. The number of agents n: We perform the experiments in the settings of n-3 and n
2. The prior distribution of states Q: In the binary case, there exists one-to-one mapping between the state w (wy- = 1111-00 W1, w)) and wh [0. 1]. which is the probability that a random reviewer votes for "accept" in the clean setting. The prior over the states can be described by a prior distribution over wi. We use three different priors for wi, which are Beta( 2. (most papers' quality is either high or low), Beta(1, 1) (papers' quality is distributed uniformly), and
papers have medium quality).
3. The bias vector b: Since the numerical experiment is to compare the scores of two papers, we consider the opposite and same biases between two papers:
opposite: Paper A has the positive bias vector [0010 where the reviewers intend to vote for "accept" without careful review. Paper has an negative bias vector b = [1.0]. This simulates situation where one paper has negative cheap signal and another paper has a positive cheap signal.
same: Both paper A and B have the same bias vector b bb= [0,1 This simulates situation where both papers have positive (negative) cheap signals.
We evaluate the performance by accuracy, which 1 minus the expected error probability, and perform the experiments when two papers have opposite biases (Figure 4 and the same bias (Figure 5). In each setting, we consider three scenarios with varying noise levels for paper A (A,=0,0.3,0.66 In each scenario, we vary the noise level of paper B and compare our Surprisal-based Score and baseline under different priors and the number of reviewers. The x axis is paper 's noise level and the y axis is the accuracy of Surprisal-basee Score (red lines) and baseline (green lines [22] According the results, our method outperforms or equals the bascline in accuracy in all cases. We also notice that our method has more advantages under the opposite bias setting and when the number of reviewers is higher.
Note that the bound is independent of the biases.
10 "max and (pmin are the abbreviations of maxxesspps) and minars:p(s) respectively. "ince the prior we choose is symmetric, for the case of b = by =[10]] the result is exactly the same.
12ehe discontinuity happens because when Aa is fixed and λS increases to certain threshold, 1 "accept's" score can be higher than 2 "accept's" score.

---

In addition, we propose another benchmark for comparison, named the SP-inspired score. This benchmark slightly modifies the original Surprisingly Popular method [2] to fit our setting. This benchmark is not invariant to noise but usually has a lower variance. We include the details and the numerical experiments comparing our method with the SP-inspired score in Appendix D
Au = 0.0
A - 0.0
(Bettl,5.33)
5..t
Baseline (Retal 5,
Baseline (Bets((5.5))
0.3
Our Method (11t
Accuree
Our Methed (Beiall.11)
Baseline (Beta(1.11)
Baseiin (Beta(L.11)
Method (.,t
Methed (Betai3.31)
Bascline (Betal3.31)
983
377
6.1
5.6
Bazeline (8eta(3,31)
02
3.1
5.5
0.8
1.0
AA 0.3
Ax
0.6
Aa
0.3
Aa - 0.6
0.3
0.5
0.7
Accurscy
0.8
0.6
0,60
02
2.4
0.5
0.6
1.0
0.2
0.4
2.5 0.6
1.0
600
02
04
2.5
0.5
0.0
02
0.4
2.5
03
1.0
Ap
Ap
(a) reviewers
(b) 5 reviewers
Figure 4: Performance evaluation (opposite biases)
= 0.0
0.0
Our Method 5.33)
Our Methed (Beta).5..5)
Beseline (Beta), 5, 511
Baseine (Beta(55.50)
Our Mathod (B.ta(1..))
Our Muthod (Betall.11)
Baseline (Betall.11)
(Betafi.11)
Our Method (3.3
Dur Method (Bwiat3.33)
Baseline (Beta)3.31)
3.7
0.1
3.6
Baseline (Beta(3,31)
03.5
02
0.1
1.0
Ap
2a
0.3
0.6
0.3
0.6
0800
22
2.4
0.5
1.0
0.2
0.4
2.5
0.6
1.2
080
02
0A
5.6
8.0
0.4
2.5
1.0
Aa
Ag
Ag
Ag
(a) 3 reviewers
(b) 5 reviewers
Figure 5: Performance evaluation (same bias)
7 Conclusion and Discussion
When papers receive ratings of different noises due to heterogeneity of paper topics and the existence of cheap signals, we propose scoring process that leads to noise-robust rank among papers. The im- plementation of the process requires the reviewers to report their predictions for other ratings but does not require any prior knowledge. We provide theoretical and numerical justification for the process.
Our theory assumes that the reviewers are perfect Bayesian. However, this may not be the case in practice. Non-Bayesian reviewers may exhibit reporting bias and prediction bias. To some extent, prediction bias is inevitable even with a rational crowd, since it is hard for reviewers to fully understand the prior. The good news is, if the prediction bias is identically distributed among reviewers, then it can be regarded as part of the noisy prior, making the solution we propose still feasible. In contrast, dealing with reporting bias in the one-shot setting appears to be more challenging, since it can violet the basic assumption of homogeneous reviewers. For instance, some reviewers may be more lenient, while others more strict. Many related works have considered this situation and proposed solutions to mitigate or calibrate the reporting error. We can employ any of these report debiasing schemes based on historical information multi-tasking, and then use our Surprisal-basee Score on the processed reports to measure the paper quality.
8 Acknowledgements
This research supported by National Key R&D Program of China (2022DD0114900)
10

---

References
[1] Drummond Rennie. Let's make peer review scientific. Nature, 535(7610):31-33, 2016.
[2] Dražen Prelec, H Sebastian Scung, and John McCoy. A solution to the single-question crowd wisdom problem. Nature, 541(763):532-533, 2017.
[3] Dražen Prelec. A Bayesian Truth Serum for subjective data. Science, 306(5695) 2004.
[4] Itai Arieli, Yakov Babichenko, and Rann Smorodinsky. Robust forecast aggregation. Proceedings of the National Academy of Sciences, 115352)E11213--12211,, 2018.
[5] Michael Mahoney. Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive therapy and research, 1(2):161-175, 1977.
[6] Charles Hess. Biases of research in the agricultural sciences. ETC: A Review of General Semantics, pages 348-353, 1977.
[7] Carole Lee, Cassidy Sugimoto, Guo Zhang, and Blaise Cronin. Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1):2-17, 2013.
[8] Andrew Tomkins, Min Zhang, and William D Heavlin. Reviewer bias in single-verse double- blind peer review. Proceedings of the National Academy of Sciences, 114(48):1278--1277 2017.
[9] Ivan Stelmakh, Nihar Shah, and Aarti Singh. On testing for biases in peer review. Advances in Neural Information Processing Systems, 32, 2019.
[10] Flaminio Squazzoni, Giangiacomo Bravo, Mike Farjam, Ana Marusic, Bahar Mehmani, Michael Willis, Aliaksandr Birukou, Pierpaolo Dondio, and Francisco Grimaldo. Peer review and gender bias: A study on 145 scholarly journals. Science advances, 7(2)eeabd0299, 2021.
[11] Ivan Stelmakh, Charvi Rastogi, Ryan Liu, Shuchi Chawla, Federico Echenique, and Nihar B Shah. Cite-seeing and reviewing: A study on citation bias in peer review. Plos one, 18(7))e0283980,22 2023.
[12] Carole Lee. Commensuration bias in peer review. Philosophy of Science, 8775):127 2015.
[13] Ivan Stelmakh, Nihar Shah, Aarti Singh, and Hal Daumé III. Prior and prejudice: The novice reviewers' bias against resubmissions in conference peer review. Proceedings of the ACM Human-Computer Interaction, ((CSCW1): 2021.
[14] Magnus Roos, Jörg Rothe, Joachim Rudolph, Björn Scheuermann, and Dietrich Stoyan. A statis- tical approach to calibrating the scores of biased reviewers: The linear vs. the nonlinear model. In Multidisciplinary Workshop Advances in Preference Handling, volume 2, page 5, 2012.
[15] Magnus Roos, Jörg Rothe, and Bjöm Scheuermann. How to calibrate the scores of biased review- by quadratic programming. In Twenty-Fifth AAAI Conference Artificial Intelligence, 2011.
[16] Emaad Manzoor and Nihar Shah. Uncovering latent biases in text: Method and application to peer review. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 35, pages 4767-4775, 2021.
[17] Weijie Su. You are the best reviewer of your papers: An owner-assisted scoring mechanism. Advances Neural Information Processing Systems, $4:27929-27995 2021.
[18] Jingyan Wang and Nihar B Shah. Your 2 is my 1, your 3 is my 9: Handling arbitrary miscalibrations in ratings. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 864-872, 2019.
[19] Siddarth Srinivasan and Jamie Morgenstern. Auctions and prediction markets for scientific peer review. arXiv preprint arXiv:1109.00923 08 2021.

---

[20] Sijun Tan, Jibang Wu, Xiaohui Bei, and Haifeng Xu. Least square calibration for peer reviews. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 27069-27088 Curran Associates, Inc., 2021.
[21] Hadi Hosseini, Debmalya Mandal, Nisarg Shah, and Kevin Shi. Surprisingly popular voting recovers rankings, surprisingly! In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 245-251. International Joint Conferences on Artificial Intelligence Organization, 8 2021. Main Track.
[22] Grant Schoenebeck and Biaoshuai Tao. Wisdom of the crowd voting:
Truthful aggregation of
voter information and preferences. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34,
pages 1872-1883. Curran Associates, Inc., 2021.
[23] Grant Schoenebeck and Fang- Yi Yu. Two strongly truthful mechanisms for three heterogeneous agents answering one question. Xujin Chen, Nikolai Gravin, Martin Hoefer, and Ruta Mehta, editors, Web and Internet Economics, pages 119-132, Cham, 2020. Springer International Publishing.
[24] David M Rothschild and Justin Wolfers. Forecasting elections: Voter intentions versus expectations. SSRN Electronic Journal, 2011.
[25] Anirban Dasgupta, Ravi Kumar, and D Sivakumar. Social sampling. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 235-243, 2012.
[26] Asa B Palley and Jack Soll. Extracting the wisdom of crowds when information shared. Management Science, (5(5):2291-2309 2019.
[27] Marcellin Martinie, Tom Wilkening, and Piers DL Howe. Using meta-predictions to identify experts in the crowd when past performance is unknown. Plos one, 15(4):e0232058, 2020.
[28] Yi-Chun Chen, Manuel Mueller-Frank, and Mallesh M Pai. The wisdom of the crowd and higher-order beliefs. arXiv preprint 1xXi::102.006660 2021.
[29] Tom Wilkening, Marcellin Martinie, and Piers DL Howe. Hidden experts in the crowd: Using meta-predictions to leverage expertise in single-question prediction problems. Management Science, 68(1):8875508 2022.
[30] Asa Palley and Ville A Satopää. Boosting the wisdom of crowds within a single judgment problem: Weighted averaging based on peer predictions. Management Science, 2023.
[31] Cem Peker. Extracting the collective wisdom in probabilistic judgments. Theory and Decision, 94(3)467-501, 2023.
[32] Robert L Winkler and Allan Murphy. "good" probability assessors. Journal of Applied Meteorology and Climatology, 7(5):751-758, 1968.
[33] Yuqing Kong and Grant Schoenebeck. Equilibrium selection in information elicitation without verification via information monotonicity, In Anna R. Karlin, editor, 9th Innovations in Theoreti- cal Computer Science Conference, ITCS 2018, January / /1-14, 2018, Cambridge, MA, USA, vol- ume 94 of LIPIcs, pages 13:1-13:20. Schloss Dagstuhl- Leibniz-Zentrum fuer Informatik, 2018.
12

---

A Detailed Examples of Our Model
Example (Binary case, clean setting). Signal 0 represents "reject" and signal 1 represents "ac- cept". There are three rypes of papers, a, 3, and 7, Type a paper's state is w" =
That is, the clean setting, a reviewer will receive "accept" signal for type a paper with probability wf = 0.2. Analogously, we let w 0.5 and wi =.88. random paper's 's state has type a, 3, and 7 with equal probability. This describes the prior distribution over paper's 's states Q:
w w.p.
W w.p.
W=
w w.p.
The prior probability that reviewer receive
"reject" signal for random paper is 90 = 0.5. The probability that both of the two review-
qo ju f + [ = 0.5, 91 receive "reject" for a paper will be Uo.0 (2)) (w))) 2 xw))) =0331. In general, the joint distribution between two reviewers' ratings is represented as a matrix:
U
T
W
100
w
w
w
81
+
w wi)
3
W1
3
1
L00
0.5
0.2
[0.80..]]
0.5,0.5]
0.2,0.8]
3
0.2
0.5
0.8
00.31
0.19
0.19
0.31
According to the joint distribution matrix U, the prediction matrix P can be established:
Uoaa
Un.
P=
Usa
Uog-U01
00.62
0..88
0.62
0.38
U1o+U1.1
U1.0-U1.1
Thus, when a reviewer receives "reject" signal, she will believe a random reviewer will receive "accept" signal with probability Po,1 0.38. When reviewer receives "accept" signal, she will believe a random reviewer will receive "accept" signal with probability P1.1=0.62.
use the same prior in Example 5 Given paper of type Y (ie. the probability that the clean signal of a reviewer is "accept" is 0.8), the paper is difficult to read so that there is 50% chance that reviewer will vote for "reject" without reading
Example (Binary case, noisy setting).
0
carefully. This shows that the paper has a noise matrix M
When the noise applies,
0.5 0.5]
MsttE,,,, En, Na, Thus, Ô Ü-MUUM thereby we obtain
0.58
0.17
Ü=M'UM=
0.17
0.08
Goa
Un.a
p
Uo.o+lo.1
Uo. UL
00.77
0.23
U1.0
0.68
0.32
U10+U1111
U2o0UU11
a reviewer receives and votes for "reject", she will predict random reviewer will receive and
vote for "accept" with probability pa,1= Po.1=0.23. If reviewer receives and votes for "accept", she
will predict a random reviewer will receive and vote for "accept" with probability P1.12
B Invariant Surprisal
We introduce a measure that is invariant to any invertible noise.
Definition& 4 (Invariant Surprisal). Given state w and joint distribution U, we define the invariant surprisal as
Surp(w, (0)(w,,)=-- (Er
where prior q the marginal distribution of U.
The invariant surprisal is closely related to the the distance between point x and distribution D by (x, = v(x-u)5 1x)--((xx-1 where
Mahalanobis distance, which measures
13

---

== == and S is the covariance matrix. The invariant surprisal is closely related to (w However, we cannot directly use d (w.Q)². In our setting, w is distribution vector where the sum of the elements is I, thus the covariance matrix S is degenerate [T3 and S is not well-defined. Thus, we modify d (w, by using the simple joint distribution matrix U rather than the covariance matrix. It is invariant and well-defined. When the number of distinct states [2]] matches Shannon's 's surprisal to some extent.
(w, ?.
Theorem 4. The invariant surprisal is non-negative, vanishes when the state w q equals the prior. and is invariant to any invertible noise. In the binary case where Σ= (0,1), the invariant surprisal Pi. is Surp w, U) (det(U))) =2)²² (go9) (P1,1 - 91)² and qo Po Pb.a
q1 Po.1+P1.0
Proof of Theorem
(w (w-))
= (w-q)M(M UM) "M-
((w-))U-\(-q)
In the binary case where Σ= accepc, recall that we use 1 for accept and 0 for reject for short. Then, can simplify the expression of surprisal in the binary case follow.
Surp(w,U) -q-
U111
-
wo- 40
=[wo-qu qu w1 -1] del(U)
-U10
U0oo
w1-91
W1
1
(wo 90) Σ Us.t)
(wy (wo-90= -w== -1 )1))
det(U)
1
(wo (wo--90²
det(U)
= (1001 (Po.o Pi,))) 12 (wo (wo-g0)²
= (909) (P, (w) -q))
The invariant surprisal is invariant to any invertible noise. However, it only reflects the distance between the state and the prior and is not suitable for many application scenarios like peer review, Nevertheless, show that when people reach a consensus in the clean state, the invariant surprisal can be used to identify the clean state. Morcover, in this case, the invariant surprisal matches Shannon's 's surprisal to some degree.
Shannon's surprisal For a random variable where x[]r[x====q) qr., Shannon's 's definition of the surprisal score of the outcome is log 1. 9= When the outcome is more likely to happen, the surprisal that it actually happens is small. To measure the surprisal of state w, we can use log 1 where (w denotes the probability that the state shows up.
dw
Example 7 (The consensus case: Invariant Surprisal Surprisal). We consider special case, call the consensus case, where agents always have the same rating. Formally there are [S]] distinct states and each state is a one-hot vector. For all the sth state's 's coordinates are all zero except that the denoted by 1,, For example, when d=3, the states are (1.0.0) (0,1,0). In this case, gs is not only the prior probability for signal s. but also the probability that the sth state shows up. Let diag( q) denote a diagonal matrix whose diagonal elements consist of vector
th coordinate
0.25
0.25
For example, is either (1,0) (0,1) with equal probability. S
0.25
0.25
14

---

U=diag(q). The invariant surprisal for the 8th state is
Surp(1
(1,
1
2++1
1
-.
9s
In the consensus case, there is one-to-one correspondence between states and signals. Our definition for the surprisal the state 1, where all agents agree to signal is 1-1= 1- which is the 1st Taylor expansion of Shannon's.
qa
In special where there are only [3] distinct states, we have the following result.
Corollary 2 (The noisy consensus case). When there only [[] distinct states W1, W2."".Wppl the invariant surprisal of each state wy is the 1st Taylor expansion of Shannon's surprisal, i.e., Surp(wk..) -1.
Qwr.
We will show that the above case is noisy version of the consensus case. The invariance directly induces the above result.
Proof of Corollary 2
When there are only [ distinct states, we can define the intrinsic noise M as a matrix where each htth row is wa such that it is the noisy version of the consensus case. Then the invariance of the invariant surprisal leads to the above corollary.
Identifying Clean State by the Invariant Surprisal Given that agents reach consensus in the clean states, we have already known the prior q and the prior unbalanced gs gr, Vs+ then we can identify the clean state in the noisy setting by the invariant surprisal. Sometimes even side information about q is sufficient. For example, the binary case, we know that the fraction of all "accept" is less than 5,, i.e., qa <0.5, then once we obtain > invariant surprisal in the noisy setting, we know the clean state is all "accept", otherwise, the clean state all "reject".
Omitted Proofs
C.1 Proof of Claim 2
Claim 2. The noisy state w = (1-A) is a convex combination of the clean state w and the bias vector b. In the binary case (E= (0(rejoct) ((cceet)))) M is the set of all non-degenerate noises where M1,1> M
Proof of Claim 2 First, show that w=(1-
W W-
(see Claim for more details) (Definition of M,,b)
-w(
=(1-A)
Then, prove that every binary M M EM is invertible and satisfies MI, M
[ 11 A+Abo
Ab
M2 = 00--111111 (1- (1=
Abo
101+Ab1
det [) M Mo,
Mr
Finally, we prove that every invertible binary noise matrix M with M1,1 Mo. belongs to M
1 Ma
Mr.1
M =
1 M1 M1,1
Ma
Ma.1
= =MM,,1 -M,1 I+ (1- M MM,1+MMa Mo,1)
1 Mo,1
Ma.
Md.
1
Mo.1
I-M
M-M
Let X 1- M1,1 + Mo.1 and b Mo.1 Ma.1
Then, we have M = (1-A)I+ AB. Because
15

---

M1.1-Mo, =de 0 and 0s Mo.1 < Mi,1 we derive that Ae [0,1). Because 1-M1,1 >0, we M
obtain that 0< 1-M1 <1. Thus, be Ag.
C.2 Proof of Claim
3
Claim 3. the binary case (Σ (O(reject), (O(reject) (accept)) the - -invariant surprisal vector can be simplified as
w-q
v V900i - Po,1)
Proof of Claim
det (U)
004 -
=oo (
=qom -Po.
By plugging in the above formula into the definition of Surp", prove the claim.
C.3 Proof of Theorem
Theorem (Invariance). The - -invariant surprisal non-negative, vanishes when the state w equals the prior q, and is invariant to MX.b for all Xe [0,1), be The implication of invariance that for any noise with Ae [0,1,) Surp" (w, USrppp'' (w,
Proof of Theorem 7 We first show that U-1 positive semi-definite, which implies the non-negativity. Afterwards, prove the invariance.
Claim 4. U and U-1 are positive semi refiniee
Proof of Claim 4 As the number of papers is finite, we denote all K states as W1. W2."" WK- where state w, represents the probability distribution of random reviewer's signal for paper i under review. Let 1 x K vector Qaæe denote the prior distribution over the states. For all s,t, U2 En (s) (t). Let W be K x|E| matrix that stacks the states, i.e. for all k the hth row of W is Wa. The above equation is equivalent to U W diag( W thus U is positive Because positive semi-definite matrix's 's inverse is also positive semi-definite. U-1 is also positive semi-definite
Claim 5. Q=qM, w = WM, Ū-M'UM.
Proof of Claim 5 When noise M applies, EI -.W, Da.eE,,,s=Easaabae
Thus, w wM, Ú = MUM. The noisy prior is the marginal distribution of the noisy joint distribution. Let 1 [xp] denote [x]]ddmmenioonl row vector with all elements equal to one. We have
-=1 x
1 1x
MM
(M' is column-stochastig
1x UM
=qM
Claim 6. det(M),b) =
16

---

Proof of Claim 6
(e((Ma))=del((1
(et B)
(del(cA)=('de(AA) where A is nxn matrix)
1-1
-(1- det (I+
1-X lbb)
X
(1- (1,1,,,,,11
=(1 =(1- (1
1-X bl')
(Sylvester's determinant theorem)
=(1 - (1-2))3(11 1-x
(b is distribution vector thus all elements sum to 1.)
=(1-1)18H1
Let w, Ú denote the noisy state and joint distribution where the noise is M=MXbb we have
Surp" "wr (.)
dett (d) मللا ( q)
=del(M UM) ألड (w
det((U) 20 (ee(MM) 1ol1 (w qQ)
det det(UU) - (Q-
(Claim 6
1-X det det(UU) マリー (w M)
(Claim 5
=det(U) 1| (w- (w-q) It A
B
1-1
=det(U) إاा (w-q) ー dut(U) ス|حي (w-
(U) शाोनग (w-q)
(wB=b=qB by definition of B)
=Surp' (w.U)
Thus, we finish the analysis for the invariant surprisal vector.
C.4 Proof of Theorem and Theorem 3
In this section, will show the omitted proof of Theorem 2 and Theorem Before proving we first analyse the results by assuming we know P in all scenarios.
Theorem and Theorem 3
Lemma (Error probability, known P). For any pair of papers A,B whose clean states are A w correspondingly (without loss generality, let E i B()), given their noise matrix A M eM correspondingly.
E Egai det((A) কगकन. 2 ())PTT det d) कपाा
/w w sexpl 22 Ew x) "e)ddttU)) E
Pr
(Fmax Pmmia) )2( (n n ==
Proof of Lemma 7 Recall that is the empirical frequency of distribution vi. When we have infinite number of reviewers, Eg bp= Ap and Ex Berpp===>> Ei. Thus,
17

---

1 a 4 B 40
B Bang
Pr
2
det (+) 2ा
det (B Iा
E Bqu
w w. E Eg
PP
det (T4) 2011-11
det dct(UB) 2013-113
Pr| [w (p- - a
(Corollary invariance)
However, when the number of reviewers is finite, we need analyse the difference between eù and è by concentration bound.
Ea Rqui
Pr
W w
det((A)) 205--1) det( (B) 2012-11
Boaip
E
B 4
Ega 4p det (ŪB) .ाo
W w
Pr
det (Q(A))) 202F
det det(B
(T)) xPF
det (A)) )
Boag-Baar
E Ew
Ess 10- Pang--aag Ban
E Pp -
W W.B
Pr
det (M) >(())11)
det BB) 2(1111
det det(UA) (+) (لل|111
det( (B) |ाل))11
Bang-BB Eaan det (04) शाड
Esprp- Egpp
E p Egl
E co- E g w .w
Pr
del (BB) शाक
(U) शा 1J
det(U (U) 20 1J
invariance of our score) (Corollary
Mang-Easpp det(( (
B up Peee---EAy 1p det(( uा det(U) (
w w
2
Esaep- Emp
Egag- E Enay
Pr
det 2 E p- Ewd ip wd .
(Claim 5
Ú=M M-M
del (Me) )
E 4- Exa
Egg Easy
Pr
Ew up p- B w wBB
1-
1-XB
(Claim 6 det(M),,) (1-2)12-11
Eqai Egpip)
Eqaip Eos
=Pr
Eww ip E up (Ewip= (. (s) w. and is the empirical frequency distribution v))
w ww
1-X1 1-
A 13
We expand 044 and " as the sum of independent random variables.
Egayp Egapp
1-X4 1-X
xnd Σ In ()))
(111)4)
(1 Σ ((27)
p(x;1)
In
1-l (1-28)
18

---

Because each x;² and x² is independent, we can define independent random variables yn (ke + BnB)
+n
((*+))
ksn
(1(1-14)
Vk =
- k-nd) (1-25)
k>n
For [,,,nn)), Smin
(111-= S8k 12 (1-aa- For &o [m 1, n (1- S S x (1-
Piin
Using the variables yk (KE [1,n +n 1), apply the standard Hoeffding's Inequality.
Bong--aam w w
Pr
det (4) ((o)-1)
det (Ú 2Cot-1)
= =Pr
in B -k=1
yk y
An
yk 2 Bw o p- Booppp-Bwaree Ewwsp/ ww
2(E End 2p)
sexp
(Hoeffding's 's Inequality)
n Umax--prinn (1-2M)
+n
Drnnx Kimi (1-A")
2( 2(Bwaig ?p)
=exp
(Pmax ?mmnn? nn(I-= (1-
Theorem 2 (Error probability, binary case). For any pair of papers A,B whose clean states are A w correspondingly (without loss of generality, let w w ), given their noise matrices MÁ M M correspondingly, the error probability Pr[S(A) S( B)] w Pr(S(A)=
(w wf' gues to 0 when the number of reviewers uA goes to infinity, and is bounded by
((())=+(11-0) +(1-0))) ( ( (1-10)) na +
(1-00))
2( wi
na(1-Aa)² mn(1-m))
where XA is the noise level of paper A and Xg is the noise level of paper 1
Proof of Theorem 2
We will show that with high probability, the prediction matrix can be constructed. Once is constructed, the result of Lemma
"Note that the bound is independent of the biases.
19

---

2 33
(1ff) (w + (1- (1- + (10)))((1((0)
(1-(1-0))) ( (1-0 ( +20 _w
(aft""
2 (105)
" +(1 -1--a1)"
(1 (1=
,uf"]
((of)" (1-
"
(1)f)))
(10f)
+1--ws))" (1 (1-1
2
BB BB
+Pr
Vdet((n))
V det((nn
{(11)))
+(1-
n
(1gg)
n
(wi
+ +(I- (1--0))""
2
2 13 -w/)
+exp
(Lemma 1
( (1-188
Theorem 3 (Error probability, general case). For any pair of tasks A,B B whose clean states are W w correspondingly (without loss of generality, let Ew w Pp), given their noise matrices M M EM* correspondingly, the error probability Pr(S(A) > ((B)(WA, w w"' ']+ (P((S(A)
S(B)/wA wB1 goes to 0 when the number of reviewers nA,BB goes infinity, and is bounded by
Σ (1-WA)ma (1-uB texp 2
seS
(Prmax Puin) ) ~ "
Proof of Theorem 3
First, we give upper bound of the probability that the score is undefined. Pr[S(A) is undefined]
=Pr[ss (ie (i
Σ Pr[Vie +s]
Σ (1- (1-10-) (1-0,,)m
xE
Similarly, Pr[S(B) is undefined] 5 E.Ex( (1 u "n"""" We have proved that with high probability, the prediction matrix can be constructed thus the scores are not undefined. We then use the result
15 Pmax and Pmin the abbreviations of max):ryp(s) and min+exy((s) respectively.
20

---

of Lemma 1
<Pr[S(A) is undefined +Pr[S( is undefined]
+P((S(A)2S(B) S(A) and SBB) are not undefined] w/ w/] ]
Σ
(1-w nd Σ
(1
Boap-Baug Bpg Bg W w
+Pr
det((AA 202--1
det (T)) 2111111
(1-w Σ (1-w
3EΣ
2(F "p xp- R وم)²
+exp
(Lemma
(9max - Pmin) ) n=(1-2a73
D SP-inspired Score
In this section, we introduce a new benchmark for comparison, inspired by the Surprising Popular (SP) mechanism, which we as the SP-inspired score. The Surprising Popular mechanism is designed to consolidate multiple reports into a single assertion of which world state we are in, rather than generating scores to compare different alternatives. Utilizing the notations explained earlier in this paper, mapping ::: represents each signal's quality, i, = A 1 =
stands for the
frequency of the noisy signal s, and d. (E, P... P represents the reconstructed prior probability of the noisy signal When dealing with binary signals (2 {0,1)), the SP mechanism presumes that the true state is 1 if di fn. otherwise, it is presumed that the true state is 0. In general settings, Q1 da
the SP mechanism determines the true state the signal that maximizes & d.
To facilitate comparisons between alternatives, we define the SP-inspired score as
Ú.
(((:).):)a((m)) >S(s)
se3. de
In the case of binary signals, we set (0) -1 and (=1 = This results in the SP-inspired score being S(( 8n
do
Difference between Surprisal-based Score and SP-inspired Score Recall our Empirical
Surprisal-based Score defined as
(Ū) -لiTo.
Both our Surprisal-based Score and the SP-inspired score involve reconstructing the prior from agents' predictions. However, they differ how to exploit this reconstructed prior to obtain a calibrated score. Our Surprisal-based Score is invariant of noise, meaning it does not change in the presence of noise when the number of reviewers is infinite. On the other hand, the SP-inspired score affected by noise. This implies that in certain situations, even when we have an infinite number of reviewers, the SP-inspired score fails to distinguish which of the two papers is better. However, our Surprisal-based Score exhibits lower stability compared the SP-inspired score, because it involves division of the correlation, which may lead to high variance when the number of reviewers is small.
Numerical Experiments We perform numerical experiments compare the performance of our Surprisal-bard Score and the SP-inspired score in the binary setting (E= (0(reject) (acceptt) )). We employ the same experimental setup as described in Section 6 Figure and Figure present the and experimental results. Upon analyzing the results, we observed that the performance of our Surprisal- based Score and the SP-inspired Score is roughly the same when the number of reviewers small.
21

---

0.0
Ag 0.0
Our Method (..5a),5.55)
Our Method (8.ta).5..5)
(B.t
(Botat1,
(Beta/1.1))
(Betail.11f
Our Method (8eto(3.31
Our Methed (Betal3,31)
(Betal3.31)
98.
(8eta13.30)
02.5
1.0
2g
0.7
0.00
0.00
000
reviewers
(b) 5 reviewers
Figure 6: Performance evaluation (opposite biases)
Aa
Method (8etal.5.55)
Our Mathed (Beia).5,51)
(8.tal,5.5)
(Bets)1.11)
(Bctail.11)
(Beta/1.11)
(Betai1.11
Our Method (Betac3.31)
Our Method (Beta13.31)
(Betw(3.31)
(Betwi3..3) 301
05
100
(a) reviewers
(b) 5 reviewers
Figure 7: Performance evaluation (same bias)
22


Reviews:
[{"review_id": "DihgyUPjrW", "reviewer": "Reviewer_SV22", "summary": "The paper tries to solve famous problem of removing bias and noise during peer-review process. The authors address the problem in one-shot setting (without historical data) and propose a novel approach that allows to remove influence of bias and noise (under some assumptions on bias and noise) on the ranking of assessed elements (reviewed papers). The approach uses scoring adjustment (used for ranking) and requires additional signal (action) from each reviewer. \nThe authors provide theoretical guarantees and small synthetic experiments.", "questions": "1. It would be nice to understand whether the assumptions are realistic (for instance, Lines 59-60 “the correct answer is positively correlated to each agent’s signal” ; Lines 82 83 “assume that the noise is positively correlated .. ”; and the other). Right now the assumptions look theoretical with no clear understanding, e,g., how frequent they are in practice.\n\n2.  The specific assumption about particular cases of noises (Lines 203-205) looks very important while being deferred to Sec3 as Definition 1 (instead of being kept in Section 2 where environment of the problem is described). It makes feeling of overclaiming in Intro, where our Results (Lines 85-91) are stated without assumption of considering invertible noise (while other assumptions and limitations are addressed before Line 85). It would be nice to have some presentation improvements to address this gap. \n\n3. The authors claim that it is nice to have the result about best ranking when the number of reviewers goes to infinity. However, I believe it is not so important result to initially stated problem, where the limitations on the number of reviewers is crucial part of the problem (in most peer review practice we deal with 3-5 reviewers, e.g., see the case of NeurIPS). So, OK to have such theorem, but this result seems not so important and relevant to the problem taken for the research. It would be nice, if the authors focus at least the presentation more on how their results help in real practical cases like n = 3-5. (Yes, I see that the authors are aware of such numbers by providing experiments with such numbers. But (a) it might be just because of simplification of experimentation; and (b) the main contribution lies in the theoretical part of the work).\n\n4. While reading from the first page till the end, I had observed changing perception (for sure, mine, subjective) of the studied setting. From the beginning of the paper till Page 6 (Lines 189-190), I thought that the setup assumes that all reviewers assess all papers, while, in practice, each agent review few numbers of papers. Sections 2.1 2.2 have made me sure that the setup is so strange. Only, Lines 189+ reverted my perception. So, I would strongly recommend to somehow help such readers as I figure out this earlier than Page 6.\n\n5.The statement in Lines 168-169 «we focus on homogeneous noise setting where the noise is the same for all reviewers and will discuss the heterogeneous setting in Section 7\" makes feeling of covering “heterogeneous setting” by this work. After reaching Section 7, I have found that Section 7 is Conclusion section and heterogeneous setting are discussed as some future work... Missed expectation\n\n========\n\nAFTER REBUTTAL\nI thank the authors for answering the questions. I hope that the answers will be reflected and fully addressed in the new revision of the work.", "limitations": "Some presentation improvements may address limitations better\nSee points 1, 2 and 3 in the field “Questions”.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-\tOverall good organization of the paper storyline (very good Intro!)\n-\tMore or less clear statements and easy to follow\n-\tSynthetic experiments", "weaknesses": "-\tSome drawbacks in clarity related to assumptions and limitations of the setup\n-\tSeem limited practical effect\n-\tThe contribution does not look very enough for NeurIPS\n-\tLack of comparison with alternative ways to improve scoring (e.g., non-1-shot ones, etc)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reviewer SV22\n\nThank you for your insightful comments and suggestions.\n\n### Q1: Whether the assumptions are realistic\n\n**Positively correlated**\n\nIn the assumptions mentioned in lines 59-60, \"positively correlated\" indicates that the correct answer has a positive correlation with the signal, as discussed in [1]. For example, if the signal is \"yes\", the posterior belief that the correct answer is \"yes\" would be greater than the prior belief. In our model (lines 82-83), the positive correlation, denoted as $M_{1,1}>M_{0,1}$, indicates that the reviewer's clean signal correlates positively with her received noisy signal. This assumption is realistic, as a negative correlation would imply that if the reviewer receives 'accept', her posterior belief regarding her clean signal (which she can ascertain with full effort) being \"accept\" would be lower than the prior belief, which is illogical. We will clarify the distinction of \"positively correlated\" in lines 59-60 and 82-83 in the final version.\n\n**Non-degenerate noise**  \n\nThe assumption of \"invertible correlation\" is prevalent in peer prediction literature [2-3]. In our paper, the assumption of \"non-degenerate noise\" in lines 82-83 (and \"invertible noise\" in line 206) means a \"invertible correlation\" between each reviewer's received signal and their clean signal. In the binary case, if $M$ is non-invertible, the reviewer's received signal and clean signal become independent, rendering the received signal uninformative. In general cases, the non-invertibility of $M$ implies that there exists two papers with distinct true qualities but receive identical evaluation distribution from reviewers. Such facts prevent the existence of an effective aggregation method when noise is non-invertible.\n\nIn the final version, we will standardize the terms \"non-degenerate noise\" and \"invertible noise\" to consistently use \"non-degenerate noise\".\n\n[1] Prelec, D., Seung, H. & McCoy, J. A solution to the single-question crowd wisdom problem. *Nature* **541**, 532–535 (2017).\n\n[2] Kong, Yuqing. \"Dominantly truthful multi-task peer prediction with a constant number of tasks.\" Proceedings of the fourteenth annual acm-siam symposium on discrete algorithms. SODA 2020.\n\n[3] Schoenebeck, Grant, and Fang-Yi Yu. \"Learning and Strongly Truthful Multi-Task Peer Prediction: A Variational Approach.\" ITCS 2021.\n\n\n### Q2: Results (Lines 85-91) are stated without assumption of considering invertible noise\n\nThe results (lines 82-91) are stated under the assumption that the noise is non-degenerate. In our paper, the term \"non-degenerate\" is synonymous with \"invertible\". In the final version, we will standardize the terms \"non-degenerate noise\" and \"invertible noise\" to consistently use \"non-degenerate noise\", and amend the phrase \"Within the model, ..\" in line 85 to \"Within the model and assumptions, ..\". Thank you for pointing this out.\n\n\n### Q3: How the results help in real practical cases like 3-5 reviewers\n\nTheorem 2 offers results for finite values of $n_A$ and $n_B$. Here, $n_A$ represents the number of reviewers for paper A, while $n_B$ represents the number of reviewers for paper B. Consequently, Theorem 2 establishes an upper bound on the error probability for realistic practical cases, such as when $n=3,5$.\n\nFor a more practical perspective, consider this: usually, we only wish to compare papers that have received at least one \"accept\" and one \"reject\" evaluation. This is because papers with unanimous 'accept' evaluations are typically accepted, while those with all 'reject' evaluations are rejected. In such scenarios, the theoretical upper bound can be simplified to: $\\exp\\left(-\\frac{2(w_1^B-w_1^A)^2}{\\frac{1}{n_A(1-\\lambda_A)^2}+\\frac{1}{n_B(1-\\lambda_B)^2}}\\right)$. Here, $w_1^B-w_1^A$ signifies the average disparity in true quality, and $\\frac{1}{n_A(1-\\lambda_A)^2}+\\frac{1}{n_B(1-\\lambda_B)^2}$ relates to the number of reviewers and the maximum bias.\n\nThis bound underscores that our methodology's error rate diminishes as the gap in true quality grows more pronounced. For example, when there are 5 reviewers with a maximum bias of 0.3, the error bound approximates to $\\exp\\left(-2.5|w_1^B-w_1^A|\\right)$. The bound is reasonable, especially given that the bias can be arbitrary. To further elucidate, in Section 6, we incorporate simulation experiments to compare our approach with the baseline.\n\n\n### Q4: Observed Changing perception of the studied setting\n\nThank you for the valuable feedback. We will specify within Example 1 that reviewers have the flexibility to assess any number of papers, and that different papers may be subject to varying biases.\n\n\n### Q5: Discussion of heterogeneous reviewer noise\n\nIn one-shot settings, tackling heterogeneous noise is challenging, primarily because the mechanism has no access to historical data from reviewers. Moreover, each reviewer typically assesses only a few papers during the review cycle. While existing research on peer grading employs Gibbs sampling to gauge the bias of each evaluator, this method necessitates a significant number of evaluations from each evaluator [4-5]. Practically, certain techniques can mitigate noise at the reviewer level. For instance, using benchmark papers can help calibrate reviewers' evaluations. In our final version of the paper, we will reintroduce the relevant discussion that was previously condensed due to page limit.\n\n[4] Piech, Chris, et al. \"Tuned models of peer assessment in MOOCs.\".\n\n[5] Mi, Fei, and Dit-Yan Yeung. \"Probabilistic graphical models for boosting cardinal and ordinal peer grading in MOOCs.\" *Proceedings of the AAAI Conference on Artificial Intelligence*. Vol. 29. No. 1. 2015."}, {"review_id": "dYw69DWL1P", "reviewer": "Reviewer_UucL", "summary": "This paper aims to detect and correct bias in Peer Review. They propose a one-shot noise calibration process without any prior information. Experiments are conducted on the binary case to show the effectiveness of the proposed method.", "questions": "1. In theorem 2, why there is 1/2 in the error probability (Pr[S(A) > S(B)|w1^A,w2^B]+1/2 Pr[S(A)=S(B)|w1^A，w1^B])? The guarantee of such an error probability may still remain a gap in the performance guarantee of the proposed method.\n2. In Sec 2.2, why “only consider the noise where M is invertible”? What will be the situation when M is non-invertible? \n3. Why can construct the vector q and the joint distribution matrix U from the prediction matrix P as stated in claim 1?  \n4. Is it universally applicable to assume that 'each individual receives the clean signal with probability 1-\\lambda’?\n", "limitations": "The authors have discussed the limitations of this paper in Sec 7.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The studied problem is important.\n2. Theoretical guarantee is provided for the proposed calibrated score.\n3. The organization of this paper is clear.\n", "weaknesses": "1. The rationality of the proposed surprisal scores needs further support. For example, in Figure 1, it seems that the left paper with lower negative reviewers’ prediction is the same as the right paper with higher negative reviewers’ prediction, while the left paper receives 1 accept and 2 reject while the right paper receives 2 accept and 1 reject. This result is confusing since the reviewers may give low P_0,1 simply due to the poor quality of the left paper.\n2. The experiments are weak in demonstrating the generality of the proposed method. For example, in Sec 6, the authors only conduct experiments on the binary case, lacking the more general settings with more types of possible signals. \n3. The theoretical guarantees lack a clear explanation. For example, in theorem 2, why there is 1/2 in the error probability (Pr[S(A) > S(B)|w1^A,w2^B]+1/2 Pr[S(A)=S(B)|w1^A，w1^B])? The guarantee of such an error probability may still remain a gap in the performance guarantee of the proposed method.\n4. Some assumption or claims lacks further support and needs to offer more clarification or explanation. For example, in Sec 2.2, why “only consider the noise where M is invertible?” What will be the situation when M is non-invertible? Why can construct the vector q and the joint distribution matrix U from the prediction matrix P as stated in claim 1?  Is it universally applicable to assume that 'each individual receives the clean signal with probability 1-\\lambda’?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reviewer UucL\n\nThank you for your insightful comments and suggestions.\n\n### W1: Rationality of the proposed surprisal scores\n\nOur understanding of your question suggests that you are seeking a more intuitive explanation of Figure 1, especially concerning why our method yield identical scores in the following two scenarios:\n\n* Scenario A: 1 accept and 2 rejects, where the 'accept' reviewer predicts $P_{1,1}=0.55$ and the 'reject' reviewers predict $P_{0,1}\\approx 0.07$.\n* Scenario B: 2 accepts and 1 reject, where the 'accept' reviewers predict $P_{1,1}=0.55$ and the 'reject' reviewer predicts $P_{0,1}\\approx 0.4$.\n\nIn Scenario A, the 'reject' reviewers anticipated that very few would vote for 'accept'. However, actually 1/3 of the reviewers vote for 'accept', making 'accept' a surprisingly popular signal. This suggests that the reviewers might have been swayed by noise skewing towards 'reject'. In contrast, Scenario 2 witnessed 'accept' votes aligning closely with the predictions, suggesting minimal interference from noise. According to our model, when a voting outcome exceeds the crowd's predictions, it is more likely to be accurate. This reasoning accounts for the identical scores assigned to both scenarios.\n\n\n### Q1(1): In theorem 2, why there is 1/2 in the error probability (Pr[S(A) > S(B)|w1^A,w2^B]+1/2 Pr[S(A)=S(B)|w1^A，w1^B])?\n\nThe occurrence of 1/2 arises when we use random selection to break ties when comparing two papers that have the same scores. For instance, a tie might arise when all reviewers vote for \"accept\" to both papers (scores being set to $+\\infty$). In these situations, there is no additional information to compare the quality of two papers, which induces 1/2 in error probability.\n\nIn real-world scenarios, we typically only compare papers that have received at least one \"accept\" and one \"reject\" evaluation. This is because papers with unanimous 'accept' evaluations are generally accepted, while those with all 'reject' evaluations are rejected. Given this practice, the likelihood of two papers garnering the same score is low.\n\n\n### Q1(2): The guarantee of such an error probability remains a gap in the performance guarantee of the proposed method.\n\nThe main reason of the gap is that our theoretical guarantees encompass all conceivable scenarios, but simulation experiments evaluate only a finite set of situations. It is crucial to underscore the importance of theoretical guarantees, as they validate the effectiveness of our mechanism. By 'effectiveness', we mean that given a sufficient number of reviewers, our method can attain arbitrarily low error rates; even with a limited number of reviewers, the error rate remains constrained and predictable.\n\n\n### Q2: Why only consider the noise where M is invertible\n\nThe assumption of \"invertible correlation\" is prevalent in peer prediction literature [1-2]. In our paper, the term \"invertible noise\" (i.e., noise matrix $M$ is invertible) signifies an \"invertible correlation\" between each reviewer's received signal and their clean signal. In the binary case, if $M$ is non-invertible, the reviewer's received signal and clean signal become independent, rendering the received signal uninformative. For general cases, the non-invertibility of $M$ implies that there exists two papers with distinct true qualities but receive the same evaluation distribution from reviewers. Such facts prevent the existence of an effective aggregation method when noise is non-invertible.\n\n[1] Kong, Yuqing. \"Dominantly truthful multi-task peer prediction with a constant number of tasks.\" Proceedings of the fourteenth annual acm-siam symposium on discrete algorithms. Society for Industrial and Applied Mathematics, 2020.\n\n[2] Schoenebeck, Grant, and Fang-Yi Yu. \"Learning and Strongly Truthful Multi-Task Peer Prediction: A Variational Approach.\" 12th Innovations in Theoretical Computer Science Conference (ITCS 2021). Schloss Dagstuhl-Leibniz-Zentrum für Informatik, 2021.\n\n\n### Q3: Why can construct the vector q and the joint distribution matrix U from the prediction matrix P as stated in claim 1\n\nClaim 1 in this paper is initially introduced by Prelec et al. in the context of the Surprisingly Popular (SP) method [3]. Here we briefly delve into the theorem's intuition. Applying Bayes' rule, we deduce $q_t=q_s\\frac{P_{s,t}}{P_{t,s}}$. Given that $\\sum_t q_t = 1$, we can solve for the prior probability of $q_s$: $q_s = (\\sum_{t}\\frac{P_{s,t}}{P_{t,s}})^{-1}$. Regarding the joint distribution $U$, by definition, we have $U_{s,t}=q_s P_{s,t}$.\n\n[3] Prelec, D., Seung, H. & McCoy, J. A solution to the single-question crowd wisdom problem. Nature 541, 532–535 (2017).\n\n\n### Q4: Is it universally applicable to assume that each individual receives the clean signal with probability 1-lambda?\n\nIn the paper we define $\\mathcal{M}^*$ as the noise family where an individual receives the clean signal with a probability of $1-\\lambda$ and a biased value (which is independent of quality) with a probability of $\\lambda$. In Claim 2, we establish that $\\mathcal{M}^*$ encompasses all positively correlated ($M_{1,1} > M_{0,1}$) and invertible noises in the context of binary signals. This emphasizes that $\\mathcal{M}^*$ serves as a universal noise model for binary signals. However, in general settings, $\\mathcal{M}^*$ is not comprehensive enough to describe all potential noise. In Appendix B, we introduce a modified metric that remains invariant across all possible invertible noises. We also demonstrate its capability in identifying the true state $\\mathbf{w}$ under specific circumstances.\n"}, {"review_id": "cF2lg5lSWO", "reviewer": "Reviewer_6t8A", "summary": "The paper considers the problem of comparing two papers based on noisy ratings, where the noise can be arbitrarily biased for different papers. The paper elicits from each reviewer both a rating and a distribution of predicted ratings from other reviewers’ (based on a Bayesian update of the common prior, known to the reviewers). The paper proposes a scoring method that recovers the correct comparison between papers from the noisy reports, which essentially corrects the reported ratings using the reported prediction distributions. The authors theoretically show a bound on the error probability of a comparison based on their scoring method, and experimentally evaluate the comparison accuracy. ", "questions": "- Could the authors expand on what makes the setting of this work unique as compared to the settings of the past works referenced in Section 1.1 (Lines 123-129)? Are none of these works applicable as a baseline for comparison?\n- The invariant surprisal vector concept is given a lot of focus (Section 3), when the purpose of this concept seems to only be that it’s used to prove Corollary 1. Since the surprisal-based score (in Section 4) is the main contribution, I might suggest making it more clear in Section 3 what the purpose of the invariant surprisal vector is, in order to keep the focus on the main contributions. \n- Could the authors clarify the aspects of Section 1 and 2 referenced above?\n", "limitations": "The authors adequately address the limitations of the assumptions made (perfect Bayesian and identical reviewers). ", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- The problem considered by the paper, calibrating biased noise in a one-shot comparison between items, is conceptually interesting. The authors motivate this problem well in the peer review setting with examples of cases where multiple reviewers may have the same bias. \n- The authors prove a theoretical bound on the convergence rate of their method’s error probability to 0. The experimental settings considered are thorough (and demonstrate aspects of the theoretical result). \n", "weaknesses": "- While the problem itself is interesting, the assumption that all agents share and know a common noisy prior is very strong; e.g., in the peer review examples in the introduction, while the noise may be biased in the same direction for all reviewers, the noisy prior may not even be similar for all reviewers. Given this assumption, I’m unsure about the significance of the proposed method, which essentially recovers this prior (as in past work) and uses it to standardize the reports.\n- I found the clarity of the writing to be generally poor, leading to some significant confusion at points (particularly in Sections 1 and 2). For example, in Section 1, I did not understand the claim following Example 2 (that a higher noise level would result in a lower expected score).  In Section 2, it was difficult to follow which aspects are observable by agents and by the mechanism.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reviewer 6t8A\n\nThank you for your insightful comments and suggestions.\n\n### W1: Assumption of common noisy prior is very strong\n\nWe interpret the scenario where different reviewers have different priors as if they share a common prior but possess distinct private information. The \"common prior assumption\" is a foundational hypothesis in economics [1]. Works that employ this assumption model agents' differences as \"information asymmetry\" rather than \"prior asymmetry\". In other words, they assume that differences among agents arise from variations in their private information, rather than differences in their prior beliefs [2].\n\n[1] Morris, Stephen. \"The common prior assumption in economic theory.\" *Economics & Philosophy* 11.2 (1995): 227-253.\n\n[2] Aumann, Robert J. \"Agreeing to Disagree.\" *The Annals of Statistics* (1976): 1236-1239.\n\n### Q1: What makes the setting of this work unique as compared to the settings of the past works\n\nIn Section 1.1, we classify related work into two primary categories: \"bias in peer review\" and \"reducing bias via second-order information\".\n\nRegarding the \"bias in peer review\" category, our objectives align with the broader goal of mitigating bias in the peer review process. However, the main distinction between ours and previous works is that our method is parameter-free and operates without historical data. In contrast, previous studies typically need to fit model parameters or access to historical data.\n\nRegarding the \"reducing bias by second-order information\" category, our study tackles a problem distinct from those addressed in previous works. They primarily concentrate on the aggregation of forecasts. The settings and methodologies of most works cannot be extended to comparing paper quality in peer review contexts.\n\nThe Surprisingly Popular (SP) method can be applied in our setting [3]. The SP method calculates the \"prediction-normalized vote\" for all signals $i$ as $\\frac{w_i}{q_i}$ and asserts that the signal $\\arg\\max_i \\frac{w_i}{q_i}$ is the best signal. Thus, when signals are binary ('reject': 0, 'accept': 1), the SP score of a paper is given by the \"prediction-normalized vote\" of 'accept' minus that of 'reject', i.e., $\\frac{w_1}{q_1}-\\frac{w_0}{q_0}$. However, the SP score is not invariant to noise. In some instances, even with infinite reviewers, the SP score fails to determine the relative quality of two papers.\n\n[3] Prelec, Dražen, H. Sebastian Seung, and John McCoy. \"A solution to the single-question crowd wisdom problem.\" *Nature* 541.7638 (2017): 532-535.\n\n**An example that the SP score fails to work**\n\n* Signals are binary ('reject'/'accept').\n* The prior over the states is $w_1\\sim \\text{Uniform}(0,1)$ (paper’s quality is distributed uniformly).\n* Paper A has true state $\\mathbf{w}_A=[1/3,2/3]$, i.e., paper A has true quality $2/3$.\n* Paper A receives no noise, i.e., the noise matrix of paper A is $\\mathbf{M}_A=\\left[\\\\begin{matrix}1 & 0\\\\\\\\0 & 1\\\\\\\\ \\\\end{matrix}\\right]$.\n* Paper B has true state $\\mathbf{w}_B=[1/4,3/4]$, i.e., paper B has true quality $3/4$, which is better than paper A.\n* Paper B receives positive noise. The noise matrix of paper B is $\\mathbf{M}_B=\\left[\\\\begin{matrix}1/3 & 2/3\\\\\\\\0 & 1\\\\\\\\ \\\\end{matrix}\\right]$.\n\nFor paper A, $\\hat{\\mathbf{w}}_A=[1/3,2/3]\\, \\hat{\\mathbf{q}}_A=[1/2,1/2]$. The SP score of paper A is $\\frac{2/3}{1/2}-\\frac{1/3}{1/2}=\\frac{2}{3}$.\n\nFor paper B, $\\hat{\\mathbf{w}}_B=[1/12,11/12]\\, \\hat{\\mathbf{q}}_B=[1/6,5/6]$. The SP score of paper B is $\\frac{11/12}{5/6}-\\frac{1/12}{1/6}=\\frac{3}{5}$.\n\nAs a result, paper B has a better true quality than paper A, but has a lower SP score, even if there are infinite reviewers. \n\n\n### Q2: Purpose of the invariant surprisal vector\n\nThe core idea is that a metric can facilitate quantitative comparisons across different papers only if it is invariant to systematically biased noise. Without invariance, comparisons for certain papers can easily be skewed by biased noise. For instance, with one noise pattern, paper A outscore paper B, while with a different biased noise pattern, paper B outscore paper A.\n\nSpecifically, the invariant surprisal vector forms the main building block for the design of the surprisal-based score. This score effectively compares papers as if we had their true quality, especially as the number of reviewers goes to infinity. This property crucially relies on the invariance. We will clarify it in the final version.\n\n\n### Q3: Could the authors clarify the aspects of Section 1 and 2\n\n**Clarification of Example 2**\n\nIn Example 2, we want to express that \"a higher noise level would result in a lower expected score for high-quality papers\", rather than \"a higher noise level would result in a lower expected score\". \n\nIn popular topics, the abundance of expert reviewers ensures a diminished noise in the review process. This ensures that the evaluations from reviewers align closely with the true paper quality. Conversely, in the topics where expert reviewers are scarce, the review process is prone to conservative noise. Consequently, there's a higher probability for reviewers' evaluations to deviate from the actual quality of the papers.\n\n**Clarification of which aspects are observable by agents and by the mechanism**\n\nIn brief, each paper has a true state $\\mathbf{w}$ representing its actual quality (nobody knows). Reviewer $i$ is aware of the prior distribution $\\mathbf{Q}$ over $\\mathbf{w}$ and receives a signal $x_i$ reflective of the paper's true state $\\mathbf{w}$. Using both the prior $\\mathbf{Q}$ and the signal $x_i$, reviewer $i$ forms a posterior distribution on $\\mathbf{w}$ to anticipate the evaluations of fellow reviewers, represented as $\\mathbf{q}_i$. The mechanism aggregates all signals and predictions $(\\mathbf{q}_i, x_i)$ to assign the paper a score $S^*$. In the final version, we will relocate the symbol clarifications (Table 1, currently in the appendix on page 17) back to section 2 for ease of reference.\n"}, {"review_id": "1KWnGthZ4f", "reviewer": "Reviewer_tv7v", "summary": "This paper proposes a method to calculate peer-reviews scores for papers in the presence of systematically biased noise, such that the score of a paper with a higher expected score in a noise-free regime is higher than the score of a paper with a lower expected score in a noise-free regime with high probability (as the number of peer reviewers grows large). The method does not use reviewers' historical scores to compute their priors. Instead, the method asks reviewers to predict the scores of other reviewers (similar to the Bayesian truth serum). Numerical experiments confirm the theoretical results that the proposed score is better able to distinguish between \"good\" and \"bad\" papers under systematically biased noise compared to a baseline that just averages the uncalibrated peer review scores.\n\n**Post-rebuttal:** I am raising my presentation score to 4 (excellent).", "questions": "Q. Could the authors report numerical results from a naive application of SP as a baseline?", "limitations": "The authors have noted the limitations of their work.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is clear and precise, and spends sufficient time setting up the problem and building intuition. The proposed method adapts a well-known idea (Bayesian truth serum) to the setting of systematically-biased noise in peer review in a principled manner; this is novel to the best of my knowledge. It is also non-trivial, and provides a useful way to construct peer-review scores without needing historical data for each peer reviewer.", "weaknesses": "The generality of the exposition also makes it a bit difficult to follow. I think just limiting the main body of the paper to the binary decision case (accept, reject) and adjusting the notation for this (eg. a for accept, r for reject) would make it easier to follow the main argument of the paper. The generalization to multi-valued decisions (eg. review scores) could be relegated to an appendix or to just one theorem.\n\nThe numerical experiments do not consider SP as a baseline (possible with some modifications since it cannot be directly applied). A naive application of SP would be the most related method in prior work for the problem considered in this paper. Hence, it would help to see how this would perform.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reviewer tv7v\n\nThank you for your insightful comments and suggestions.\n\n### W1: Suggestion for limiting the main body of the paper to the binary decision case:\n\nThank you for your suggestion. We will adopt it in the final version. This allows us to focus the main body on the binary signal case, making it easier for readers to follow the paper's flow and understand the core concepts.\n\n### Q1: Could the authors report numerical results from a naive application of SP as a baseline\n\nThe Surprisingly Popular (SP) method can be applied in our setting [1]. The SP method calculates the \"prediction-normalized vote\" for all signals $i$ as $\\frac{w_i}{q_i}$ and asserts that the signal $\\arg\\max_i \\frac{w_i}{q_i}$ is the best signal. Thus, when signals are binary ('reject': 0, 'accept': 1), the SP score of a paper is given by the \"prediction-normalized vote\" of 'accept' minus that of 'reject', i.e., $\\frac{w_1}{q_1}-\\frac{w_0}{q_0}$. However, the SP score is not invariant to noise. In some instances, even with infinite reviewers, the SP score fails to determine the relative quality of two papers (a specific example is provided below).\n\n[1] Prelec, Dražen, H. Sebastian Seung, and John McCoy. \"A solution to the single-question crowd wisdom problem.\" *Nature* 541.7638 (2017): 532-535.\n\n**An example that the SP score fails to work**\n\n* Signals are binary ('reject'/'accept').\n* The prior over the states is $w_1\\sim \\text{Uniform}(0,1)$ (paper’s quality is distributed uniformly).\n* Paper A has true state $\\mathbf{w}_A=[1/3,2/3]$, i.e., paper A has true quality $2/3$.\n* Paper A receives no noise, i.e., the noise matrix of paper A is $\\mathbf{M}_A=\\left[\\begin{matrix}1 & 0\\\\\\\\0 & 1\\\\\\\\\\\\end{matrix}\\right]$.\n* Paper B has true state $\\mathbf{w}_B=[1/4,3/4]$, i.e., paper B has true quality $3/4$, which is better than paper A.\n* Paper B receives positive noise because its writing is good. The noise matrix of paper B is $\\mathbf{M}_B=\\left[\\begin{matrix}1/3 & 2/3\\\\\\\\0 & 1\\\\\\\\\\\\end{matrix}\\right]$.\n\nFor paper A, we can calculate that $\\hat{\\mathbf{w}}_A=[1/3,2/3]\\, \\hat{\\mathbf{q}}_A=[1/2,1/2]$. Thus the SP score of paper A is $\\frac{2/3}{1/2}-\\frac{1/3}{1/2}=\\frac{2}{3}$.\n\nFor paper B, we can calculate that $\\hat{\\mathbf{w}}_B=[1/12,11/12]\\, \\hat{\\mathbf{q}}_B=[1/6,5/6]$. Thus the SP score of paper B is $\\frac{11/12}{5/6}-\\frac{1/12}{1/6}=\\frac{3}{5}$.\n\nAs a result, paper B has a better true quality than paper A, but has a lower SP score, even if there are an infinite number of reviewers.\n"}, {"review_id": "oF70IsiwVO", "reviewer": "Reviewer_VrqZ", "summary": "The paper tackles a problem in peer review where reviewers may provide noisy/biased ratings for papers. The paper investigate a \"one-shot\" scoring process inspired by the \"Surprisingly Popular\" method, that can rank papers by their true quality without any prior knowledge, even if different papers have different noise levels and biases.\n\nTheir method relies on eliciting reviewers’ predictions for a random reviewer’s signal and using this for calibration. Specifically, their model assumes that there is a prior distribution over the states (where each paper's state in the \"clean\" setting is the distribution of reviewers’ signals)  and that reviewers are perfect Bayesians who can update their beliefs based on their private signals. In the \"noisy\" setting, reviewers can only observe noisy versions of their signals, determined by a random mapping that depends on the noise level and the bias vector.\n\nThen, the paper defines the true quality of a paper as the expected score in the clean setting, and aims to design a method that can rank papers by their true quality with high probability, even if they have different noises/biases. For this, they introduce a surprisal vector that measures how \"surprising\" a paper’s state is compared to the prior distribution, and is normalized by the correlation between reviewers’ signals. The key result is that the surprisal vector is noise-invariant, i.e., the surprisal vector is the same in the clean and the noisy setting, regardless of the noise model.  Then, by computing an empirical surprise-based score for each paper, the mechanism papers can be compared in a noise-invariant way and are consistent with their true qualities.", "questions": "- How do you handle det(U) being negative in practice?\n\n- Can you give some intuition for why is the normalization of the surprisal vector important?\n\n- Do the exponential bounds in the general case not depend on the cardinality of the signal space? Can you give some intuition for why this is the case? \n\n- Are the results robust if noise in agents' predictions of other agents' signals correlated with the noise in their own signals?", "limitations": "The main limitation I see is that it is unclear what to do if some of the computed statistics (like det(U)) violate the non-negativity requirement.\n\nThe authors are up front about other limitations. These aren't ideal, but may be OK for the goals of this paper.\n- Authors assume no incentive issues, i.e., that all agents report truthfully. However, this is unlikely to be the case in practice, especially if participants know that there is \n\n- The authors assume that agents are calibrated in their predictions, even if their signals may be biased. This does feel like a bit of an odd assumption, and is unlikely to be true. Why would signals be biased, but predictions of others' signals be calibrated? ", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper is tackling an important problem, which is the question of how to de-bias and de-noise reviewers to obtain a clearer view of paper quality.\n\n- The observation that the surprisal vector is invariant under noise is insightful and interesting, as was the idea to look to the surprisingly popular mechanism.", "weaknesses": "- The presentation of the paper could be improved. The intro feels a bit unfocused, there are a number of grammatical errors, and the paper is a bit notation heavy. It might help to move the notation table from the appendix to the main body, and also expand on the \"high level ideas\" to more provide a clearer overview of the proposed method. It took a couple reads to understand exactly what was being proposed. \n\n- The assumption of calibrated agent predictions is not clearly justifiable in a context where agent signals are assumed to be potentially biased.\n\n- In line 162, the result cited by Kong et al. (2018) is outdated, suggest updating with more recent results that improve on this:\n\nSchoenebeck, Grant, and Fang-Yi Yu. \"Two strongly truthful mechanisms for three heterogeneous agents answering one question.\" ACM Transactions on Economics and Computation 10.4 (2023): 1-26.\n\nSrinivasan, Siddarth, and Jamie Morgenstern. \"Auctions and prediction markets for scientific peer review.\" arXiv preprint arXiv:2109.00923 (2021).\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reviewer VrqZ\n\nThank you for your insightful comments and suggestions.\n\n### W1: The presentation of the paper could be improved\n\nIn the final version, we will relocate the symbol clarifications (Table 1, currently in the appendix on page 17) back to section 2 for ease of reference. Besides, we will provide a more detailed and clear overview in \"high level ideas\". Thank you for valuable suggestions.\n\n\n### W3: Outdated result cited by Kong et al. (2018)\n\nThanks for pointing out, we will update these references in the final version. In addition, we will make the related work section more detailed in the final version.\n\n\n### Q1: How do you handle det(U) being negative in practice\n\nIn our model, when reviewers are Bayesian, det(U) will never become negative. In practice, a negative det(U) suggests that when a reviewer favors a paper, her belief in the likelihood of another reviewer also favoring it decreases. This unusual scenario signals to the Program Committee that this particular paper warrants further discussion and careful consideration before making a final decision.\n\n\n### Q2: Can you give some intuition for why is the normalization of the surprisal vector important\n\nRecall that in the binary case, our score is defined as (baseline score - prior expected scores) / correlation.\n\nFirst, the normalization constant adaptively compensates for varying noise levels. When faced with noisy reviewers providing weak signals beyond the prior, the baseline and prior expected scores will closely align, resulting in a correlation near zero. By normalizing using this small correlation, the overall scores are scaled up to compensate for the strong noise and weak signals. Conversely, when noise is weak and the reviewers' signals are strong, there is a clear distinction between the baseline and prior expected scores, leading to a higher correlation. This adaptive normalization ensures the metric's efficacy across contexts with varying noise levels.\n\nSecond, only with the normalization, we can have the invariance results. The core idea is that a metric can only enable quantitative comparisons across different papers if and only if it is invariant to systematically biased noise. Without invariance, biased noise can easily distort comparisons for certain papers. For instance, with one noise pattern, paper A outscore paper B, while with a different biased noise pattern, paper B outscore paper A.\n\n\n### Q3: Do the exponential bounds in the general case not depend on the cardinality of the signal space? Can you give some intuition for why this is the case\n\nIn general cases, there exists a \"natural\" mapping $\\varphi$ that allocates a real-valued score to each potential signal. For example, in NeurIPS, this mapping is represented as $r \\rightarrow 3, br \\rightarrow 4, ba \\rightarrow 5, wa \\rightarrow 6, a \\rightarrow 7$. Such a \"natural\" mapping is prevalent in most peer review conferences. By employing this mapping, we can apply the concentration bound on the average mapping outcome, which is expressed solely in terms of the value range $\\varphi_{\\text{max}}-\\varphi_{\\text{min}}$.\n\n\n### Q4: Are the results robust if noise in agents' predictions of other agents' signals correlated with the noise in their own signals\n\nOur assumption in the paper is that the reviewers' biases towards a paper consistently reflect in their predictions about others. If your question is, \"Is the result robust if this correlation is not perfect Bayesian?\", then the response would be that our score loses its theoretical guarantee in such cases. Nonetheless, our scores of the papers still provide meaningful insights for the Program Committee, particularly when assessing papers of borderline quality.\n\n\n### L1: Authors assume no incentive issues\n\nThe research question this paper attempts to address is \"how to aggregate the biased evaluations in peer review, assuming evaluations are truthful\". This research question is parallel and complementary to the research question \"how to design incentive mechanisms to obtain truthful evaluations\". We can employ mechanisms aimed at eliciting truthful evaluations (for example, the papers you mentioned) to ensure truthfulness, and then use our proposed method for aggregation. We will clarify this in the discussion section in the final version.\n\n\n### L2: Why would signals be biased, but predictions of others' signals be calibrated\n\nIn our paper, we interpret the scenario where a reviewer provides biased evaluation as she may not exert sufficient effort and only have access to \"cheap signals\". However, she can still report calibrated predictions concerning other reviewers’ evaluation, based on the signal she has. This scenario mirrors the Surprisingly Popular (SP) method [1]. In SP, it is assumed that while agents might give a biased response to a question like \"what is the capital of Illinois\", they can still provide a Bayesian prediction regarding other people’s answers based on their own response.\n\n[1] Prelec, Dražen, H. Sebastian Seung, and John McCoy. \"A solution to the single-question crowd wisdom problem.\" *Nature* 541.7638 (2017): 532-535.\n"}]

Author rebuttal:
We thank the reviewers for their insightful suggestions and comments on our manuscript. We are pleased that the reviewers recognize the novelty (Reviewer SV22, tv7v, VrqZ), clarity in writing and organization (Reviewer SV22, UucL, tv7v), as well as the sound theoretical guarantees and synthetic experiments (Reviewer SV22, UucL, 6t8A). We will address the inquiries and comments from each reviewer separately.

Meta-review: This paper proposes a method to mitigate bias and noise during peer-review process.
The authors present theoretical guarantees in a simplified setting, accompanied by synthetic experiments.

Reviewers agree that the paper is very well-motivated. The topic is important, and has been gaining traction recently, especially in light of the state of reviewing in the machine learning community. The primary concern reviewers have about this paper lie in the strength of the assumptions. However, this topic is relatively young, and so it benefits from this type of simplified analysis, to better understand the problem parameters in tractable settings. I thus recommend acceptance. I believe this paper will be an insightful step in the quest for better peer review mechanisms.

I encourage the authors to take into account the feedback from reviewers in the camera-ready. Specifically, several reviewers noted that the later sections are overly general, with unclear payoff for the reader – and I tend to agree.
Decision: Accept (poster)