Title: Calibrating “Cheap Signals” in Peer Review without a Prior
Venue: NeurIPS 2023
Accepted: yes
Keywords: Peer prediction

Peer review

Calibration
Abstract: Peer review lies at the core of the academic process, but even well-intentioned reviewers can still provide noisy ratings. While ranking papers by average ratings may reduce noise, varying noise levels and systematic biases stemming from ``cheap'' signals (e.g. author identity, proof length) can lead to unfairness. 
Detecting and correcting bias is challenging, as ratings are subjective and unverifiable. Unlike previous works relying on prior knowledge or historical data, we propose a one-shot noise calibration process without any prior information. We ask reviewers to predict others' scores and use these predictions for calibration. Assuming reviewers adjust their predictions according to the noise, we demonstrate that the calibrated score results in a more robust ranking compared to average ratings, even with varying noise levels and biases.
In detail, we show that the error probability of the calibrated score approaches zero as the number of reviewers increases and is significantly lower compared to average ratings when the number of reviewers is small.

Calibrating 'Cheap Signals' in Peer Review without a Prior

Abstract

Peer review lies at the core of the academic process, but even well-intentioned reviewers can still provide noisy ratings. While ranking papers by average ratings may reduce noise, varying noise levels and systematic biases stemming from 'cheap' signals (e.g. author identity, proof length) can lead to unfairness. Detecting and correcting bias is challenging, as ratings are subjective and unverifiable. Unlike previous works relying on prior knowledge or historical data, we propose a one-shot noise calibration process without any prior information. We ask reviewers to predict others' scores and use these predictions for calibration. Assuming reviewers adjust their predictions according to the noise, we demonstrate that the calibrated score results in a more robust ranking compared to average ratings, even with varying noise levels and biases. In detail, we show that the error probability of the calibrated score approaches zero as the number of reviewers increases and is significantly lower compared to average ratings when the number of reviewers is small.

1 Introduction

Peer review is fundamental to the scientific process. However, as noted in Rennie [1]:

But it is a human system. Everybody involved brings prejudices, misunderstandings, and gaps in knowledge, so no one should be surprised that peer review is often biased and inefficient ... even with the best of intentions ...

Let us consider the following scenario.

Example 1 (Peer review) . We have two papers and each paper is reviewed by 5 reviewers. We can only accept one paper. The first paper receives 4 'accept' and 1 'reject', the second paper receives 3 'accept'and 2 'reject'.

A typical choice is to accept the first paper since it receives more fraction of 'accept'. Ideally, we should accept the paper which receives more fraction of 'accept' in expectation from reviewers who have full expertise and invest full efforts, called the clean setting. However, in practice, the ratings can be noisy due to the lack of expertise or effort. If the noise is consistent for the two papers, the better paper should still receive more fraction of 'accept' in expectation. Nevertheless, the noise can be different for different papers.

Example 2 (Hot vs. cold topic) . The topic of the first paper is more popular than the second paper. In this case, the first paper is easier to obtain reviewers with high expertise. Thus, in the noisy setting, the first paper has less noisy ratings than the second paper.

It is possible that in the clean setting, the first paper receives 80% of 'accept' in expectation and the second paper receives 90%. However, in the noisy setting, the first paper may still obtain 80% 'accept' in expectation while the second paper may only obtain 60% 'accept' in expectation because it receives more noisy ratings. In addition to the noise level, the noise can be systematically biased due to the existence of 'cheap signals'.

Example 3 (Long proof vs. short proof) . The first paper has a complicated proof. The second paper has a short proof. In the noisy setting, each reviewer (unconsciously) intends to vote for 'accept' for paper with a long proof, even if the length of the proof is a 'cheap signal'.

It may occur in practice that the second paper is better in the clean setting, but the first paper receives more fraction of 'accept' in expectation in the noisy setting, due to the systematically biased noise caused by cheap signals. Cheap signals can also be the authors' reputation, institutions, or writing skills. One may argue that the cheap signals may correlate with expensive signals, which represent the true quality of the paper. However, these cheap signals like proof length can be easily manipulated.

Prior knowledge and historical data may help to calibrate the noise. However, the identification of bias can be subjective and ad hoc. It will be more applicable if we do not need prior knowledge and design a general one-shot process. A fundamental question arises:

* Can we design a one-shot scoring process that leads to a noise-robust rank without any prior knowledge? Formally, we want the paper with a higher expected score in the clean setting also have a higher score in the noisy setting with high probability, even if different papers' reviews have different noise levels and biases.

A key challenge is that the ratings are subjective and unverifiable. As an outsider without any prior knowledge, we do not even know whether the bias exists or not. Therefore, it is impossible to address the above question without eliciting any additional information from the reviewers. Prelec [2] propose an innovative method, Surprisingly Popular (SP), to aggregate the information despite agents' systematic biases without any prior. Here each agent is asked to report both her signal (e.g. accept or reject) and her prediction (e.g. 90% reject, 10% accept) for a random peer's report. The elicited predictions are used to construct the prior. Instead of a majority vote, the aggregation is the signal which is more popular compared to the prior 2 . By assuming that the ground truth is positively correlated to each agent's signal 3 , the aggregation is correct when the size of the crowds is sufficiently large.

We adopt the signal-prediction framework of SP, i.e., asking each reviewer to additionally provide her prediction for a randomly selected reviewer's report. Besides, we follow the literature of information elicitation [3] and information aggregation [4] to assume that agents are perfect Bayesian, which establishes a start for theoretical analysis:

Assumption 1 (Bias generated from 'cheap signals' can be predicted) . (Informal) We assume agents are perfect Bayesian. Besides, when their ratings are noisy and have systematic biases, they will adjust their predictions based on the biased noise.

The above assumption essentially provides a general definition of the bias. With this assumption, we aim to design a one-shot scoring process with the reviewers' predictions as calibrations.

We cannot directly apply SP to this scenario, not only because it requires a large number of agents, but also because it is designed to solve a problem with objective ground truth. In the peer review setting, each paper's ratings are subjective and do not need to be the same even in the clean setting. We may ask the reviewer to compare two papers and apply SP to aggregate the comparisons. However, in practice, papers are reviewed independently by a possibly different set of reviewers.

Moreover, it is possible that both two papers' surprisingly popular ratings are 'accept'. To rank them, one idea is to compare the 'amount' of surprise. For example, SP calibrates the ratings by dividing the prior, where the prior is constructed by the predictions. Subtracting the prior is another option. However, neither of the two calibration options is robust to the noise. In Example 2, when the prior is 50% / 50% for both the clean setting and the noisy setting, both of the two papers' surprisingly popular ratings are 'accept'. However, the second paper's 'accept' is less popular compared to the prior because its ratings have more noise, while the second paper is better in the clean setting.

We formulate the above problem into a formal mathematical model, defining a paper's true quality as its expected rating in the clean setting without noise, and assuming that the noise is non-degenerate and each reviewer's received noisy signal positively correlates with her clean signal 4 .

Our Results Within the model and assumptions, we propose a one-shot scoring process. When the number of reviewers is infinite, a paper with a higher score in the noisy setting will also have a higher true quality even if the papers have different biased noises. When the number of reviewers is finite, we provide a bound for the error probability and numerically show that our method beats the baseline, the average rating, in various situations. We also extend the results to non-binary setting naturally, with proper restrictions on the noise structures. Of an independent interest, we also provide surprisal measures that are invariant to noises.

High-level Ideas The baseline score is the average rating, e.g., the fraction of 'accept' votes. In this paper, we define the Surprisal-based Score in the form of baseline score -prior expected score correlation between ratings , where the correlation is measured by a power of the determinant of the joint distribution matrix between two reviewers' ratings. Intuitively, the subtraction of the prior will cancel the effect of the systematic bias, and the division of the correlation will compensate for varying noise levels across contexts. For example, when faced with noisy reviewers providing weak signals beyond the prior, the average rating and the prior will closely align, resulting in a small correlation between ratings. By normalizing using this small correlation, the surprisal score is scaled up to compensate for the strong noise.

Three Reviewers, Binary Ratings To illustrate our scoring process, we provide an example for the case of three reviewers, binary ratings.

Example 4 (Three reviewers, binary ratings) . In the setting of binary ratings, the signal of reviewers is either 'accept' (1) or 'reject' (0). We call the reviewer who votes for 'accept' as positive reviewer and the reviewer who votes for 'reject' as negative reviewer. We map 3 'accept' signals to the highest score, and 3 'reject' signals to the lowest score. When there are at least an 'accept' and a 'reject', we construct a matrix ˆ P , where ˆ P s,t is the average prediction, from the reviewers who report signal s , for the probability that a random reviewer votes for signal t . For example, ˆ P 0 , 1 is the average of the negative reviewers' predictions for the probability that a random reviewer votes for 'accept'.

Specifically, the score is defined as

Figure 1 illustrates how the score depends on the reviewers' predictions. At a high level, the score will be higher when the reviewers underestimate the probability that a random reviewer rates 'accept' (the lower left corner) 5 .

To the best of our knowledge, we are the pioneer to rigorously formulate the noise-robust comparison problem with a one-shot scoring process, when different rating tasks have different noises. We believe our work provides a start in the peer review setting for calibrating the bias without any prior.

1.1 Related Work

Bias in Peer Review Studies on peer review originated with Mahoney [5] and Hess [6]. Since then, numerous studies have emerged to identify sources and impacts of bias in peer review [7, 8, 9, 10, 11, 12, 13]. In addition to exploring sources and impacts, recent studies

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Negative reviewer's prediction

P

0, 1

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Positive reviewer's prediction

P

1, 1

1 accept and 2 reject

32

16

8

4

2

1

0

1

2

4

8

16

32

Score

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Negative reviewer's prediction

P

0, 1

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Positive reviewer's prediction

P

1, 1

2 accept and 1 reject

32

16

8

4

2

1

0

1

2

4

8

16

32

Score

have attempted to detect or calibrate biases automatically, employing various methods such as optimization, nonparametric estimation, and mechanism design [14, 15, 16, 17, 18, 19, 20]. However, these methods rely on historical data or a reviewer's votes for multiple papers.

Mechanism Design via Second-order Information Prelec [3] introduced the signal-prediction framework where agents are asked to provide both their answers and predictions. By applying the signal-prediction framework, researchers have discovered methods to aggregate people's answers to the surprisingly popular option [2], reduce bias by agents' predictions about ranking [21], incentivise truthful agents[3, 22, 23], and find more informative answers [24, 25]. In addition to the discrete setting, recently, a growing literature [26, 27, 28, 29, 30, 31] has extended the signal-prediction framework to forecast aggregation settings. In this work, we employ second-order information for noise-robust comparison in a different setting.

2 Background and Problem Statement

2.1 Background: Single-task Information Elicitation

In this section, we describe how agents receive private signals and generate predictions, following the standard signal-prediction framework [2, 3].

Model We regard each paper as a task with a state w ∈[ 0 , 1 ] 1 ×∣ Σ ∣ , where Σ is the set of all possible signals . There are n agents (reviewers) assigned to the task 6 , labeled from 1 to n . Each agent i ∈ [ n ] receives private signal X i = s with probability w s . The notation x i is the realization of X i , representing the signal actually received by agent i . In this paper, we use the notation s or t to refer to a particular signal and the notation i or j to refer to a particular agent. There is a prior distribution Q over the states and W is a random state drawn from the prior distribution Q . We use a row vector q ∈[ 0 , 1 ] 1 ×∣ Σ ∣ to denote the prior probability that an agent will receive each signal, i.e., for all s ∈ Σ , q s = Pr Q [ X i = s ]=∑ w Pr Q [ W = w ] w s . Analogously, P ∈[ 0 , 1 ] ∣ Σ ∣×∣ Σ ∣ denotes the prediction matrix such that P s,t = Pr Q [ X j = t ∣ X i = s ] ,i ≠ j is the probability that agent j will receive the signal t conditioning on another agent i receives the signal s . The joint distribution matrix U ∈[ 0 , 1 ] ∣ Σ ∣×∣ Σ ∣ is the matrix where ∀ s,t ∈ Σ ,U s,t = q s P s,t = Pr Q [ X i = s,X j = t ] 7 . Because agents' identities do not matter, we omit i,j ∈[ n ] in the definition. To avoid degeneration, we assume the determinant of U is non-zero. In the task, each agent i is asked to report her signal s i (e.g. accept) and her prediction p i for a random agent's signal (e.g. 40% accept, 60% reject) without any communication to other agents.

The above model describes how agents receive signals without noise , which we call the clean setting. Figure 2 illustrates the model briefly and we offer a detailed example in Appendix A. We will introduce the noise model later.

w = [0.2,0.8]

1

13

令田

1

1

-3

-3

0.2

0.8

Rej

Acc

Model Inference For each agent i ∈[ n ] , given her private signal s i , we assume that her prediction p i is the Bayesian posterior, i.e., ∀ t ∈ Σ , p i ( t ) = P s i ,t . If for all signal s ∈ Σ , there exists an agent who reports s , then we can obtain the prediction matrix P according to the agents' predictions. We can use P to induce the joint distribution matrix U where ∀ s,t ∈ Σ ,U s,t = q s P s,t = Pr Q [ X i = s,X j = t ] .

Claim 1 (Prelec et al. [2]) . We can construct the vector q and the joint distribution matrix U from the prediction matrix P : ∀ s,t ∈ Σ ,q s =(∑ t P s,t P t,s ) -1 ( 0 / 0 ≡ 0 ) ,U s,t = q s P s,t .

In this paper, we will assume the reviewers have good intentions and will tell the truth. That is, if she receives an 'accept' signal for a paper, she will vote for 'accept' as well. Regarding incentive design, prediction reports can be incentivized by proper scoring rules [32]. For the signal report, Prelec [3] develops a payment method called Bayesian Truth Serum (BTS) to incentivize agents to tell the truth when the number of agents is infinite. By assuming that agents perform the same strategy, Kong et al. [33] propose a method that only requires the number of agents greater or equal 6 .

2.2 Problem Statement

Noise Model In the noisy setting, instead of observing the original signal s , which we call the clean signal , each agent can only observe its noisy version M ( s ) , which we call the noisy signal , where M ∶ Σ ↦ Σ is a random operator. Denote M as the matrix where M s,t is the probability of observing noisy signal t given clean signal s . We only consider the non-degenerate noise whose noise matrix M is invertible. Additionally, we focus on the homogeneous noise which corresponds to systematic biases. Homogeneous noise means that the noise is the same for all reviewers of a specific paper.

In the noisy setting, we use the notation ˆ Q to denote the prior distribution over the noisy states. Similar to the clean setting, ˆ X i denotes the random variable of agent i 's noisy signal and ˆ x i is its realization; ˆ U denotes the joint distribution matrix where ˆ U s,t = Pr ˆ Q [ ˆ X i = s, ˆ X j = t ] ; ˆ w denotes the noisy state; ˆ q denotes the noisy prior over signals and ˆ P denotes the noisy prediction matrix. Note that the relationship between ˆ P , ˆ U and ˆ q remains the same. By abusing notation a little bit, we sometimes write ˆ Q = M ( Q ) , ˆ U = M ( U ) , ˆ w = M ( w ) . Figure 3 briefly introduces how noise affects reviewers' ratings, and we offer a detailed example of noise model in Appendix A.

Assumption 2. In the noisy setting, we assume that for each agent i , given her private noisy signal s i , her prediction ˆ p i is still the Bayesian posterior where ∀ t ∈ Σ , ˆ p i ( t )= Pr ˆ Q [ X j = t ∣ X i = s i ]= ˆ P s i ,t .

Ideally, we want to reconstruct the original state w from the noisy ˆ w and ˆ P . However, we do not have a sufficient amount of information. Instead, we aim to solve the following problem.

Noise-Robust Comparison We have two rating tasks, say reviews for paper A and B, whose clean states are w A and w B respectively. Both w A and w B follow the distribution Q . There is a known one-to-one mapping φ ∶ Σ → R between signals in Σ and paper scores. For example, when ratings are binary, the mapping φ is { reject → 0 , accept → 1 } .

w = [0.2,0.8]

W=[0.6,0.4]

noisy

clean

1

0

M:

=

0.5

0.5]

0.6

0.4

Rej

Acc

w =[0.8,0.2]

W =[0.4,0.6]

noisy

clean

0.5

0.5]

M

=

0

1

0.4

0.6

Rej

Acc

(a) Negative bias of reviewers' ratings of a paper with state w =[ 0 . 2 , 0 . 8 ] : there is a 50% chance that a reviewer will vote for 'reject' without reading it carefully. In this case, a random reviewer of this paper will vote for 'accept' with probability 0 . 4 .

(b) Positive bias of reviewers' ratings of a paper with state w =[ 0 . 8 , 0 . 2 ] : there is a 50% chance that a reviewer will vote for 'accept' without reading it carefully. In this case, a random reviewer of this paper will vote for 'accept' with probability 0 . 6 .

Therefore, for simplicity, we sometimes write signal s as a score of real number and Σ as a set of scores. For example, in the binary case, Σ ={ 0 ( reject ) , 1 ( accept )} . We care about the true quality which is defined as the expected score in the clean state, i.e., E w φ =∑ s φ ( s ) w s . For example, in the binary case, the quality is E w φ = w 1 , i.e., the probability of a random reviewer voting for 'accept'.

Paper A has n A reviewers and paper B has n B reviewers. They do not need to have the same number of reviewers. The pair ( ˆ x A i , ˆ p A i ) i ∈[ n A ] denotes the reviewers' actual ratings and predictions for paper A, and ( ˆ x B j , ˆ p B j ) j ∈[ n B ] analogously. Let S (⋅) be a function that takes reviewers' noisy ratings and predictions of a single paper as input, and outputs a calibrated score for the paper. For a pair of papers, we will rank the paper according to their calibrated scores 8 . Given a set of noises M , we aim to design S (⋅) such that for all M A , M B ∈ M , the error probability Pr [ S (⋅) ranks A higher than B ∣ B's quality is better than A's ] is upper-bounded and the upper bound goes to 0 when both n A and n B go to infinity.

3 Invariant Surprisal Vector

This section will introduce the key ingredient of our scoring process, a surprisal vector which is invariant to a natural family of noises. We first introduce this family of noises. Here each individual receives the clean signal with probability 1 -λ > 0 . With probability λ , which we call noise level, each individual receives a signal according to a distribution vector b ∈ ∆ Σ , which we call bias. We state the noise formally in the following definition.

Definition 1 (A Family of Noises M ∗ ) . We consider a family of noises with two parameters, a noise level λ ∈ [ 0 , 1 ) and a bias vector b ∈ ∆ Σ , where ∆ Σ is the probability simplex on the signal set Σ . Let B denote the matrix whose rows are all b . The noise is defined as M λ, b =( 1 -λ ) I + λ B .

Claim 2. The noisy state ˆ w =( 1 -λ ) w + λ b is a convex combination of the clean state w and the bias vector b . In the binary case ( Σ ={ 0 ( reject ) , 1 ( accept )} ), M ∗ is the set of all non-degenerate noises where M 1 , 1 > M 0 , 1 .

Then, we define a surprisal vector that is invariant to the above noises.

Definition 2 ( M ∗ -Invariant Surprisal Vector) . Given paper's state w and joint distribution U , we define the M ∗ -invariant surprisal vector as where q ∈[ 0 , 1 ] 1 ×∣ Σ ∣ denotes the marginal distribution of U .

Notation Description

Claim 3. In the binary case ( Σ = { 0 ( reject ) , 1 ( accept )} ), the M ∗ -invariant surprisal vector can be simplified as

Theorem 1 (Invariance) . The M ∗ -invariant surprisal is non-negative, vanishes when the state w equals the prior q , and is invariant to M λ, b for all λ ∈[ 0 , 1 ) , b ∈ ∆ Σ . The implication of invariance is that for any noise with λ ∈[ 0 , 1 ) , Surp ∗ ( w , U )= Surp ∗ ( ˆ w , ˆ U ) .

The proofs of Claim 2, Claim 3 and Theorem 1 are deferred to Appendix C. For general noise, the direction of w -q may not be invariant. For example, one possible noise is flipping the ratings, i.e., rating 'accept' when 'reject' and rating 'reject' when 'accept'. In this case, the direction of w -q will be flipped as well. However, if we only care about the amount of surprisal, we successfully design a measure that is invariant to all possible non-degenerate noise and show that it can be used to identify the true state w in some scenarios (see Appendix B).

4 Surprisal-based Score

This section introduces our scoring process based on the surprisal vector introduced in Section 3. We first introduce the score based on 'wishful thinking', i.e., when we have the state (or the noisy state). We then resolve this 'wishful thinking' by providing an estimation of the state.

Definition 3 (Surprisal-based Score) . Given w and U , we define the M ∗ -invariant Surprisal-based Score as

According to the simplification in the binary case (Claim 3), the M ∗ -invariant surprisal score in the binary case can be simplified to

Theorem 1 directly implies the following results: given the noisy states, we can compare their Surprisal-based Scores S ∗ ( w , U ) and the comparison is invariant to the noise in M ∗ and consistent to the comparison of their true qualities.

Corollary 1 (Noise-invariant comparison by S ∗ (⋅) ) . For any two states w A , w B which follow distribution Q , and any two noises M A ,M B ∈M ∗ ,

When there are only a small number of reviewers, we will use an estimation of the Surprisal-based Score, which we call the Empirical Surprisal-based Score. We first focus on the binary case. We have already proved that for any invertible noise matrix M where M 1 , 1 > M 0 , 1 , the metric w 1 -q 1 √ ( q 0 q 1 ( P 1 , 1 -P 0 , 1 )) can be used to implement a noise-invariant comparison (Claim 2, Corollary 1). We will use the reviewers' ratings and predictions to estimate the score. We employ a frequency vector ˆ v ∈ [ 0 , 1 ] 1 ×∣ Σ ∣ to denote the frequency of the reviewers' reported signals. Formally, for all s ∈ Σ , ˆ v s = 1 n ∑ n i = 1 1 [ ˆ x i = s ] , where ˆ x i is the reported noisy signal of reviewer i . For example, in the binary case, ˆ v =( ˆ v 0 , ˆ v 1 ) where ˆ v 1 is the fraction of 'accept' and ˆ v 0 is the fraction of 'reject'.

Empirical Surprisal-based Score (Binary) When there exists at least one 'accept' and one 'reject', we construct a 2 × 2 prediction matrix ˆ P based on the reviewers' ratings and predictions. Each element of the matrix, ˆ P s,t , is the average prediction from the reviewers who report the signal s , for the probability that a random reviewer reports the signal t . For example, ˆ P 0 , 1 is the average of the negative reviewers' prediction for the probability a random reviewer votes for 'accept'. When reviewers' ratings are the same, the matrix P cannot be constructed. In this case, we set +∞ score for the papers that all reviewers vote for 'accept' and -∞ for the papers that all reviewers vote for 'reject'. Formally,

⎩ where ˆ q 1 = ˆ P 1 , 1 ˆ P 0 , 1 + ˆ P 1 , 0 and ˆ q 0 = ˆ P 1 , 0 ˆ P 0 , 1 + ˆ P 1 , 0 (See Claim 1 for the calculation of ˆ q from ˆ P ).

We naturally extend the process to non-binary settings. For example, review signals contain multiple grades ( Σ ={-2 ( reject ) , -1 ( weak reject ) , 1 ( weak accept ) , 2 ( accept )} ).

Empirical Surprisal-based Score (General) When ˆ v s > 0 ∀ s ∈ Σ , we construct the prediction matrix ˆ P . The calibrated score is defined as where ∀ s ∈ Σ , ˆ q s =(∑ t ˆ P s,t ˆ P t,s ) -1 ( 0 / 0 ≡ 0 ). When det ( ˆ U )< 0 , S (⋅) remains undefined. This 'bad event' arises when a reviewer favors a paper, her belief in the likelihood of another reviewer also favoring it decreases. This unusual scenario alerts the Program Committee that this particular paper warrants further discussion and careful consideration before making a final decision. When comparing tasks with undefined scores, our method degenerates to the baseline, i.e., comparing the value of E ˆ v φ .

5 Theoretical Guarantee

In this section, we theoretically analyse the performance of our Empirical Surprisal-based Score. We use the error probability to measure the performance. Recall that we use the expected score of a paper in the clean state to measure its true quality. In the binary case, the true quality of a paper, whose clean state is w , is w 1 , i.e., the fraction of 'accept' ratings in the clean setting. The error probability is the probability that the score ranks paper A higher than paper B while B's quality is better than A's.

Theorem 2 (binary) and Theorem 3 (general) show the theoretical upper bound of the error probability of our method. When the number of reviewers goes to infinity, the error probability goes to zero. The analysis follows from the results of Corollary 1 and a standard concentration bound analysis. We defer the proofs of these two theorems to Appendix C.

Theorem 2 (Error probability, binary case) . For any pair of papers A,B whose clean states are w A , w B correspondingly (without loss of generality, let w A 1 < w B 1 ), given their noise matrices

M A , M B ∈ M ∗ correspondingly, the error probability Pr [ S ( A ) > S ( B )∣ w A 1 ,w B 1 ] + 1 2 Pr [ S ( A ) = S ( B )∣ w A 1 ,w B 1 ] goes to 0 when the number of reviewers n A ,n B goes to infinity, and is bounded by

where λ A is the noise level of paper A and λ B is the noise level of paper B 9 .

The error bound has three terms. The first two terms bound the error probability conditioning on at least one of the papers having an infinite score, i.e., every reviewer votes for 'accept' ('reject'). The last term bounds the error probability conditioning on both papers has at least an 'accept' and a 'reject' vote. This theorem shows that we can achieve a better error bound when the rating difference ( w B 1 -w A 1 ) between two papers is larger, when there are more reviewers ( n A and n B ), and when the noise levels ( λ A and λ B ) are lower.

Theorem 3 (Error probability, general case) . For any pair of tasks A,B whose clean states are w A , w B correspondingly (without loss of generality, let E w A φ < E w B φ ), given their noise matrices M A , M B ∈ M ∗ correspondingly, the error probability Pr [ S ( A ) > S ( B )∣ w A , w B ] + 1 2 Pr [ S ( A ) = S ( B )∣ w A , w B ] goes to 0 when the number of reviewers n A ,n B goes to infinity, and is bounded by 10

6 Numerical Experiments

We perform numerical experiments to compare the performance of our Surprisal-based Score and the baseline score in the binary setting ( Σ ={ 0 ( reject ) , 1 ( accept )} ). Recall that the baseline is the proportion of the 'accept' ratings. Here we describe the parameters we select in numerical experiments.

The number of agents n : We perform the experiments in the settings of n = 3 and n = 5 .

The bias vector b : Since the numerical experiment is to compare the scores of two papers, we consider the opposite and same biases between two papers:

The prior distribution of states Q : In the binary case, there exists a one-to-one mapping between the state w = ( w 0 = 1 -w 1 ,w 1 ) and w 1 ∈ [ 0 , 1 ] , which is the probability that a random reviewer votes for 'accept' in the clean setting. The prior over the states can be described by a prior distribution over w 1 . We use three different priors for w 1 , which are Beta ( 1 2 , 1 2 ) (most papers' quality is either high or low), Beta ( 1 , 1 ) (papers' quality is distributed uniformly), and Beta ( 3 , 3 ) (most papers have a medium quality).

opposite : Paper A has the positive bias vector b A =[ 0 , 1 ] where the reviewers intend to vote for 'accept' without careful review. Paper B has an negative bias vector b B =[ 1 , 0 ] . This simulates a situation where one paper has a negative cheap signal and another paper has a positive cheap signal.

same : Both paper A and B have the same bias vector b A = b B =[ 0 , 1 ] 11 . This simulates a situation where both papers have positive (negative) cheap signals.

Weevaluate the performance by accuracy, which is 1 minus the expected error probability, and perform the experiments when two papers have opposite biases (Figure 4) and the same bias (Figure 5). In each setting, we consider three scenarios with varying noise levels for paper A ( λ A = 0 , 0 . 3 , 0 . 6 ). In each scenario, we vary the noise level of paper B and compare our Surprisal-based Score and baseline under different priors and the number of reviewers. The x axis is paper B's noise level and the y axis is the accuracy of Surprisal-based Score (red lines) and baseline (green lines) 12 . According to the results, our method outperforms or equals the baseline in accuracy in all cases. We also notice that our method has more advantages under the opposite bias setting and when the number of reviewers is higher.

In addition, we propose another benchmark for comparison, named the SP-inspired score. This benchmark slightly modifies the original Surprisingly Popular method [2] to fit our setting. This benchmark is not invariant to noise but usually has a lower variance. We include the details and the numerical experiments comparing our method with the SP-inspired score in Appendix D.

0.0

0.2

0.4

0.6

0.8

1.0

B

0.5

0.6

0.7

0.8

A

=0.0

0.0

0.2

0.4

0.6

0.8

1.0

B

0.5

0.6

0.7

0.8

A

=0.3

0.0

0.2

0.4

0.6

0.8

1.0

B

0.5

0.6

0.7

0.8

A

=0.6

Our Method (Beta(.5,.5))

Baseline (Beta(.5,.5))

Our Method (Beta(1,1))

Baseline (Beta(1,1))

Our Method (Beta(3,3))

Baseline (Beta(3,3))

Our Method (Beta(.5,.5))

Baseline (Beta(.5,.5))

Our Method (Beta(1,1))

Baseline (Beta(1,1))

Our Method (Beta(3,3))

Baseline (Beta(3,3))

(a)

3

reviewers

0.0

0.2

0.4

0.6

0.8

1.0

B

0.5

0.6

0.7

0.8

0.9

A

=0.0

0.0

0.2

0.4

0.6

0.8

1.0

B

0.5

0.6

0.7

0.8

0.9

A

=0.3

0.0

0.2

0.4

0.6

0.8

1.0

B

0.5

0.6

0.7

0.8

0.9

A

=0.6

Our Method (Beta(.5,.5))

Baseline (Beta(.5,.5))

Our Method (Beta(1,1))

Baseline (Beta(1,1))

Our Method (Beta(3,3))

Baseline (Beta(3,3))

Our Method (Beta(.5,.5))

Baseline (Beta(.5,.5))

Our Method (Beta(1,1))

Baseline (Beta(1,1))

Our Method (Beta(3,3))

Baseline (Beta(3,3))

(b)

5

reviewers

0.0

0.2

0.4

0.6

0.8

1.0

B

0.5

0.6

0.7

0.8

A

=0.0

0.0

0.2

0.4

0.6

0.8

1.0

B

0.5

0.6

0.7

0.8

A

=0.3

0.0

0.2

0.4

0.6

0.8

1.0

B

0.5

0.6

0.7

0.8

A

=0.6

Our Method (Beta(.5,.5))

Baseline (Beta(.5,.5))

Our Method (Beta(1,1))

Baseline (Beta(1,1))

Our Method (Beta(3,3))

Baseline (Beta(3,3))

Our Method (Beta(.5,.5))

Baseline (Beta(.5,.5))

Our Method (Beta(1,1))

Baseline (Beta(1,1))

Our Method (Beta(3,3))

Baseline (Beta(3,3))

(a)

3

reviewers

0.0

0.2

0.4

0.6

0.8

1.0

B

0.5

0.6

0.7

0.8

0.9

A

=0.0

0.0

0.2

0.4

0.6

0.8

1.0

B

0.5

0.6

0.7

0.8

0.9

A

=0.3

0.0

0.2

0.4

0.6

0.8

1.0

B

0.5

0.6

0.7

0.8

0.9

A

=0.6

Our Method (Beta(.5,.5))

Baseline (Beta(.5,.5))

Our Method (Beta(1,1))

Baseline (Beta(1,1))

Our Method (Beta(3,3))

Baseline (Beta(3,3))

Our Method (Beta(.5,.5))

Baseline (Beta(.5,.5))

Our Method (Beta(1,1))

Baseline (Beta(1,1))

Our Method (Beta(3,3))

Baseline (Beta(3,3))

(b)

5

reviewers

7 Conclusion and Discussion

When papers receive ratings of different noises due to heterogeneity of paper topics and the existence of cheap signals, we propose a scoring process that leads to a noise-robust rank among papers. The implementation of the process requires the reviewers to report their predictions for other ratings but does not require any prior knowledge. We provide theoretical and numerical justification for the process.

Our theory assumes that the reviewers are perfect Bayesian. However, this may not be the case in practice. Non-Bayesian reviewers may exhibit reporting bias and prediction bias. To some extent, prediction bias is inevitable even with a rational crowd, since it is hard for reviewers to fully understand the prior. The good news is, if the prediction bias is identically distributed among reviewers, then it can be regarded as part of the noisy prior, making the solution we propose still feasible. In contrast, dealing with reporting bias in the one-shot setting appears to be more challenging, since it can violet the basic assumption of homogeneous reviewers. For instance, some reviewers may be more lenient, while others are more strict. Many related works have considered this situation and proposed solutions to mitigate or calibrate the reporting error. We can employ any of these report debiasing schemes based on historical information or multi-tasking, and then use our Surprisal-based Score on the processed reports to measure the paper quality.

References

Drummond Rennie. Let's make peer review scientific. Nature , 535(7610):31-33, 2016.

Dražen Prelec, H Sebastian Seung, and John McCoy. A solution to the single-question crowd wisdom problem. Nature , 541(7638):532-535, 2017.

Dražen Prelec. A Bayesian Truth Serum for subjective data. Science , 306(5695):462-466, 2004.

Itai Arieli, Yakov Babichenko, and Rann Smorodinsky. Robust forecast aggregation. Proceedings of the National Academy of Sciences , 115(52):E12135-E12143, 2018.

Michael J Mahoney. Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive therapy and research , 1(2):161-175, 1977.

Charles E Hess. Biases of research in the agricultural sciences. ETC: A Review of General Semantics , pages 348-353, 1977.

Carole J Lee, Cassidy R Sugimoto, Guo Zhang, and Blaise Cronin. Bias in peer review. Journal of the American Society for Information Science and Technology , 64(1):2-17, 2013.

Andrew Tomkins, Min Zhang, and William D Heavlin. Reviewer bias in single-versus doubleblind peer review. Proceedings of the National Academy of Sciences , 114(48):12708-12713, 2017.

Ivan Stelmakh, Nihar Shah, and Aarti Singh. On testing for biases in peer review. Advances in Neural Information Processing Systems , 32, 2019.

Flaminio Squazzoni, Giangiacomo Bravo, Mike Farjam, Ana Marusic, Bahar Mehmani, Michael Willis, Aliaksandr Birukou, Pierpaolo Dondio, and Francisco Grimaldo. Peer review and gender bias: A study on 145 scholarly journals. Science advances , 7(2):eabd0299, 2021.

Ivan Stelmakh, Charvi Rastogi, Ryan Liu, Shuchi Chawla, Federico Echenique, and Nihar B Shah. Cite-seeing and reviewing: A study on citation bias in peer review. Plos one , 18(7):e0283980, 2023.

Carole J Lee. Commensuration bias in peer review. Philosophy of Science , 82(5):1272-1283, 2015.

Ivan Stelmakh, Nihar B Shah, Aarti Singh, and Hal Daumé III. Prior and prejudice: The novice reviewers' bias against resubmissions in conference peer review. Proceedings of the ACM on Human-Computer Interaction , 5(CSCW1):1-17, 2021.

Magnus Roos, Jörg Rothe, Joachim Rudolph, Björn Scheuermann, and Dietrich Stoyan. A statistical approach to calibrating the scores of biased reviewers: The linear vs. the nonlinear model. In Multidisciplinary Workshop on Advances in Preference Handling , volume 2, page 5, 2012.

Magnus Roos, Jörg Rothe, and Björn Scheuermann. How to calibrate the scores of biased reviewers by quadratic programming. In Twenty-Fifth AAAI Conference on Artificial Intelligence , 2011.

Emaad Manzoor and Nihar B Shah. Uncovering latent biases in text: Method and application to peer review. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 4767-4775, 2021.

Weijie Su. You are the best reviewer of your own papers: An owner-assisted scoring mechanism. Advances in Neural Information Processing Systems , 34:27929-27939, 2021.

Jingyan Wang and Nihar B Shah. Your 2 is my 1, your 3 is my 9: Handling arbitrary miscalibrations in ratings. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems , pages 864-872, 2019.

Siddarth Srinivasan and Jamie Morgenstern. Auctions and prediction markets for scientific peer review. arXiv preprint arXiv:2109.00923 , 08 2021.

Sijun Tan, Jibang Wu, Xiaohui Bei, and Haifeng Xu. Least square calibration for peer reviews. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 27069-27080. Curran Associates, Inc., 2021.

Hadi Hosseini, Debmalya Mandal, Nisarg Shah, and Kevin Shi. Surprisingly popular voting recovers rankings, surprisingly! In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 , pages 245-251. International Joint Conferences on Artificial Intelligence Organization, 8 2021. Main Track.

Grant Schoenebeck and Biaoshuai Tao. Wisdom of the crowd voting: Truthful aggregation of voter information and preferences. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 1872-1883. Curran Associates, Inc., 2021.

Grant Schoenebeck and Fang-Yi Yu. Two strongly truthful mechanisms for three heterogeneous agents answering one question. In Xujin Chen, Nikolai Gravin, Martin Hoefer, and Ruta Mehta, editors, Web and Internet Economics , pages 119-132, Cham, 2020. Springer International Publishing.

David M Rothschild and Justin Wolfers. Forecasting elections: Voter intentions versus expectations. SSRN Electronic Journal , 2011.

Anirban Dasgupta, Ravi Kumar, and D Sivakumar. Social sampling. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 235-243, 2012.

Asa B Palley and Jack B Soll. Extracting the wisdom of crowds when information is shared. Management Science , 65(5):2291-2309, 2019.

Marcellin Martinie, Tom Wilkening, and Piers DL Howe. Using meta-predictions to identify experts in the crowd when past performance is unknown. Plos one , 15(4):e0232058, 2020.

Yi-Chun Chen, Manuel Mueller-Frank, and Mallesh M Pai. The wisdom of the crowd and higher-order beliefs. arXiv preprint arXiv:2102.02666 , 2021.

Tom Wilkening, Marcellin Martinie, and Piers DL Howe. Hidden experts in the crowd: Using meta-predictions to leverage expertise in single-question prediction problems. Management Science , 68(1):487-508, 2022.

Asa B Palley and Ville A Satopää. Boosting the wisdom of crowds within a single judgment problem: Weighted averaging based on peer predictions. Management Science , 2023.

Cem Peker. Extracting the collective wisdom in probabilistic judgments. Theory and Decision , 94(3):467-501, 2023.

Robert L Winkler and Allan H Murphy. 'good' probability assessors. Journal of Applied Meteorology and Climatology , 7(5):751-758, 1968.

Yuqing Kong and Grant Schoenebeck. Equilibrium selection in information elicitation without verification via information monotonicity. In Anna R. Karlin, editor, 9th Innovations in Theoretical Computer Science Conference, ITCS 2018, January 11-14, 2018, Cambridge, MA, USA , volume 94 of LIPIcs , pages 13:1-13:20. Schloss Dagstuhl - Leibniz-Zentrum fuer Informatik, 2018.

A Detailed Examples of Our Model

Example 5 (Binary case, clean setting) . Signal 0 represents 'reject' and signal 1 represents 'accept'. There are three types of papers, α , β , and γ . Type α paper's state is w α =[ w α 0 ,w α 1 ]=[ 0 . 8 , 0 . 2 ] . That is, in the clean setting, a reviewer will receive 'accept' signal for type α paper with probability w α 1 = 0 . 2 . Analogously, we let w β 1 = 0 . 5 and w γ 1 = 0 . 8 . A random paper's state has type α , β , and γ with equal probability. This describes the prior distribution over paper's states Q :

.

⎩ The prior probability that a reviewer receive 'reject' signal for a random paper is q 0 = 1 3 w α 0 + 1 3 w β 0 + 1 3 w γ 0 = 0 . 5 , q 1 = 1 -q 0 = 0 . 5 . The probability that both of the two reviewers receive 'reject' for a paper will be U 0 , 0 = 1 3 ( w α 0 ) 2 + 1 3 ( w β 0 ) 2 + 1 3 ( w γ 0 ) 2 = 0 . 31 . In general, the joint distribution between two reviewers' ratings is represented as a matrix:

According to the joint distribution matrix U , the prediction matrix P can be established:

Example 6 (Binary case, noisy setting) . We use the same prior in Example 5. Given a paper of type γ (i.e. the probability that the clean signal of a reviewer is 'accept' is 0 . 8 ), the paper is difficult to read so that there is a 50% chance that a reviewer will vote for 'reject' without reading

⎣ ⎦ Thus, when a reviewer receives 'reject' signal, she will believe a random reviewer will receive 'accept' signal with probability P 0 , 1 = 0 . 38 . When a reviewer receives 'accept' signal, she will believe a random reviewer will receive 'accept' signal with probability P 1 , 1 = 0 . 62 .

it carefully. This shows that the paper has a noise matrix M =[ 1 0 0 . 5 0 . 5 ] . When the noise applies,

∀ s,t ∈ Σ , ˆ U s,t =∑ a,b U a,b M a,s M b,t . Thus, ˆ U = M ⊺ UM , thereby we obtain

⎣ ⎦ If a reviewer receives and votes for 'reject', she will predict a random reviewer will receive and vote for 'accept' with probability ˆ P 0 , 1 = 0 . 23 . If a reviewer receives and votes for 'accept', she will predict a random reviewer will receive and vote for 'accept' with probability ˆ P 1 , 1 = 0 . 32 .

B Invariant Surprisal

We introduce a measure that is invariant to any invertible noise.

Definition 4 (Invariant Surprisal) . Given state w and joint distribution U , we define the invariant surprisal as where prior q is the marginal distribution of U .

The invariant surprisal is closely related to the affine-invariant Mahalanobis distance, which measures the distance between a point x and a distribution D by d M ( x , D ) = √ ( x -µ ) S -1 ( x -µ ) where

µ = E [ x ] and S is the covariance matrix. The invariant surprisal is closely related to d M ( w ,Q ) 2 . However, we cannot directly use d M ( w ,Q ) 2 . In our setting, w is a distribution vector where the sum of the elements is 1, thus the covariance matrix S is degenerate 13 and S -1 is not well-defined. Thus, we modify d M ( w ,Q ) 2 by using the simple joint distribution matrix U rather than the covariance matrix. It is invariant and well-defined. When the number of distinct states is ∣ Σ ∣ , it matches Shannon's surprisal to some extent.

Theorem 4. The invariant surprisal is non-negative, vanishes when the state w = q equals the prior, and is invariant to any invertible noise. In the binary case where Σ ={ 0 , 1 } , the invariant surprisal is Surp ( w , U ) = ( det ( U )) -1 ( w 1 -q 1 ) 2 = ( q 0 q 1 ( P 1 , 1 -P 0 , 1 )) -1 ( w 1 -q 1 ) 2 and q 0 = P 1 , 0 P 0 , 1 + P 1 , 0 , q 1 P 0 , 1 P + P .

Proof of Theorem 4.

In the binary case where Σ ={ reject , accept } , recall that we use 1 for accept and 0 for reject for short. Then, we can simplify the expression of surprisal in the binary case as follow.

The invariant surprisal is invariant to any invertible noise. However, it only reflects the distance between the state and the prior and is not suitable for many application scenarios like peer review. Nevertheless, we show that when people reach a consensus in the clean state, the invariant surprisal can be used to identify the clean state. Moreover, in this case, the invariant surprisal matches Shannon's surprisal to some degree.

Shannon's surprisal For a random variable X where ∀ x, Pr [ X = x ] = q x , Shannon's definition of the surprisal score of the outcome x is log 1 q x . When the outcome is more likely to happen, the surprisal that it actually happens is small. To measure the surprisal of a state w , we can use log 1 q w where q w denotes the probability that the state w shows up.

Example 7 (The consensus case: Invariant Surprisal ≈ Surprisal) . We consider a special case, call it the consensus case, where agents always have the same rating. Formally there are ∣ Σ ∣ distinct states and each state is a one-hot vector. For all s , the s th state's coordinates are all zero except that the s th coordinate is 1, denoted by 1 s . For example, when d = 3 , the states are ( 1 , 0 , 0 ) , ( 0 , 1 , 0 ) , ( 0 , 0 , 1 ) . In this case, q s is not only the prior probability for signal s , but also the probability that the s th state shows up. Let diag ( q ) denote a diagonal matrix whose diagonal elements consist of vector

q . U diag q . The invariant surprisal for the s th state is

In the consensus case, there is a one-to-one correspondence between states and signals. Our definition for the surprisal of the state 1 s where all agents agree to signal s is 1 q s -1 = 1 q 1 s -1 which is the 1st Taylor expansion of Shannon's.

In a special case where there are only ∣ Σ ∣ distinct states, we have the following result.

Corollary 2 (The noisy consensus case) . When there are only ∣ Σ ∣ distinct states w 1 , w 2 , /uni22EF , w ∣ Σ ∣ , the invariant surprisal of each state w k is the 1st Taylor expansion of Shannon's surprisal, i.e., Surp ( w k , P )= 1 q w k -1 .

We will show that the above case is a noisy version of the consensus case. The invariance directly induces the above result.

Proof of Corollary 2. When there are only ∣ Σ ∣ distinct states, we can define the intrinsic noise M as a matrix where each k th row is w k such that it is the noisy version of the consensus case. Then the invariance of the invariant surprisal leads to the above corollary.

Identifying Clean State by the Invariant Surprisal Given that agents reach consensus in the clean states, if we have already known the prior q and the prior is unbalanced q s ≠ q t , ∀ s ≠ t , then we can identify the clean state in the noisy setting by the invariant surprisal. Sometimes even side information about q is sufficient. For example, in the binary case, if we know that the fraction of all 'accept' is less than .5, i.e., q a < 0 . 5 , then once we obtain > 1 invariant surprisal in the noisy setting, we know the clean state is all 'accept', otherwise, the clean state is all 'reject'.

C Omitted Proofs

C.1 Proof of Claim 2

Claim 2. The noisy state ˆ w =( 1 -λ ) w + λ b is a convex combination of the clean state w and the bias vector b . In the binary case ( Σ ={ 0 ( reject ) , 1 ( accept )} ), M ∗ is the set of all non-degenerate noises where M 1 , 1 > M 0 , 1 .

Proof of Claim 2. First, we show that ˆ w 1 λ w λ b

=( -) + . ˆ w = wM λ, b (see Claim 5 for more details) = w (( 1 -λ ) I + λ B ) (Definition of M λ, b )

Then, we prove that every binary M = M λ, b ∈M ∗ is invertible and satisfies M 1 , 1 > M 0 , 1 .

Finally, we prove that every invertible binary noise matrix M with M 1 , 1 > M 0 , 1 belongs to M ∗ .

Let λ 1 M 1 , 1 M 0 , 1 and b 1 M 0 , 1 λ , M 0 , 1 λ . Then, we have M 1 λ I λ B . Because

M 1 , 1 -M 0 , 1 = det ( M )≠ 0 and 0 ≤ M 0 , 1 < M 1 , 1 ≤ 1 , we derive that λ ∈[ 0 , 1 ) . Because 1 -M 1 , 1 > 0 , we obtain that 0 < M 0 , 1 1 -M 1 , 1 + M 0 , 1 < 1 . Thus, b ∈ ∆ Σ .

C.2 Proof of Claim 3

Claim 3. In the binary case ( Σ = { 0 ( reject ) , 1 ( accept )} ), the M ∗ -invariant surprisal vector can be simplified as

Proof of Claim 3.

By plugging in the above formula into the definition of Surp ∗ , we prove the claim.

C.3 Proof of Theorem 1

Theorem 1 (Invariance) . The M ∗ -invariant surprisal is non-negative, vanishes when the state w equals the prior q , and is invariant to M λ, b for all λ ∈[ 0 , 1 ) , b ∈ ∆ Σ . The implication of invariance is that for any noise with λ ∈[ 0 , 1 ) , Surp ∗ ( w , U )= Surp ∗ ( ˆ w , ˆ U ) .

Proof of Theorem 1. We first show that U -1 is positive semi-definite, which implies the non-negativity. Afterwards, we prove the invariance.

Claim 4. U and U -1 are positive semi-definite.

Proof of Claim 4. As the number of papers is finite, we denote all K states as w 1 , w 2 , /uni22EF , w K , where state w i represents the probability distribution of a random reviewer's signal for paper i under review. Let 1 × K vector q state denote the prior distribution over the states. For all s,t , U s,t =∑ k q w k w k ( s ) w k ( t ) . Let W be a K ×∣ Σ ∣ matrix that stacks the states, i.e., for all k , the k th row of W is w k . The above equation is equivalent to U = W ⊺ diag ( q state ) W thus U is positive semi-definite. Because a positive semi-definite matrix's inverse is also positive semi-definite. U -1 is also positive semi-definite.

Claim 5. ˆ q = qM , ˆ w = wM , ˆ U = M ⊺ UM .

Proof of Claim 5. When noise M applies, ∀ t ∈ Σ , ˆ w t =∑ s w s M s,t , ∀ s,t ∈ Σ , ˆ U s,t =∑ a,b U a,b M a,s M b,t . Thus, ˆ w = wM , ˆ U = M ⊺ UM . The noisy prior is the marginal distribution of the noisy joint distribution. Let 1 1 ×∣ Σ ∣ denote a ∣ Σ ∣ -dimensional row vector with all elements equal to one. We have

Claim 6. det ( M λ, b )=( 1 -λ ) ∣ Σ ∣-1

Proof of Claim 6.

Let ˆ w , ˆ U denote the noisy state and joint distribution where the noise is M = M λ, b , we have

Thus, we finish the analysis for the invariant surprisal vector.

C.4 Proof of Theorem 2 and Theorem 3

In this section, we will show the omitted proof of Theorem 2 and Theorem 3. Before proving Theorem 2 and Theorem 3, we first analyse the results by assuming we know ˆ P in all scenarios.

Lemma 1 (Error probability, known ˆ P ) . For any pair of papers A,B whose clean states are w A , w B correspondingly (without loss of generality, let E w A φ < E w B φ ), given their noise matrix M A , M B ∈M ∗ correspondingly,

Proof of Lemma 1. Recall that ˆ v is the empirical frequency of distribution ˆ w . When we have infinite number of reviewers, E ˆ v A φ = E ˆ w A φ and E ˆ v B φ = E ˆ w B φ . Thus,

However, when the number of reviewers is finite, we need to analyse the difference between ˆ w and ˆ v by concentration bound.

We expand ˆ v A and ˆ v B as the sum of independent random variables.

Because each x A i and x B j is independent, we can define independent random variables y k ( k ∈[ 1 ,n A + n B ]) as

For k ∈ [ 1 ,n A ] , φ min n A ( 1 -λ A ) ≤ y k ≤ φ max n A ( 1 -λ A ) . For k ∈ [ n A + 1 ,n A + n B ] , -φ max n B ( 1 -λ B ) ≤ y k ≤-φ min n B ( 1 -λ B ) . Using the variables y k ( k ∈[ 1 ,n A + n B ]) , we can apply the standard Hoeffding's Inequality.

(Hoeffding's Inequality)

Theorem 2 (Error probability, binary case) . For any pair of papers A,B whose clean states are w A , w B correspondingly (without loss of generality, let w A 1 < w B 1 ), given their noise matrices M A , M B ∈ M ∗ correspondingly, the error probability Pr [ S ( A ) > S ( B )∣ w A 1 ,w B 1 ] + 1 2 Pr [ S ( A ) = S ( B )∣ w A 1 ,w B 1 ] goes to 0 when the number of reviewers n A ,n B goes to infinity, and is bounded by

where λ A is the noise level of paper A and λ B is the noise level of paper B 14 .

Proof of Theorem 2. We will show that with high probability, the prediction matrix can be constructed. Once it is constructed, we use the result of Lemma 1.

Theorem 3 (Error probability, general case) . For any pair of tasks A,B whose clean states are w A , w B correspondingly (without loss of generality, let E w A φ < E w B φ ), given their noise matrices M A , M B ∈ M ∗ correspondingly, the error probability Pr [ S ( A ) > S ( B )∣ w A , w B ] + 1 2 Pr [ S ( A ) = S ( B )∣ w A , w B ] goes to 0 when the number of reviewers n A ,n B goes to infinity, and is bounded by 15

Proof of Theorem 3. First, we give an upper bound of the probability that the score is undefined.

Similarly, Pr [ S ( B ) is undefined ] ≤ ∑ s ∈ Σ ( 1 -ˆ w B s ) n B . We have proved that with high probability, the prediction matrix can be constructed thus the scores are not undefined. We then use the result

D SP-inspired Score

In this section, we introduce a new benchmark for comparison, inspired by the Surprising Popular (SP) mechanism, which we term as the SP-inspired score. The Surprising Popular mechanism is designed to consolidate multiple reports into a single assertion of which world state we are in, rather than generating scores to compare different alternatives. Utilizing the notations explained earlier in this paper, mapping φ ∶ Σ → R represents each signal's quality, ˆ v s = 1 n ∑ n i = 1 1 [ ˆ x i = s ] stands for the frequency of the noisy signal s , and ˆ q s =(∑ t ˆ P s,t ˆ P t,s ) -1 represents the reconstructed prior probability of the noisy signal s . When dealing with binary signals ( Σ ={ 0 , 1 } ), the SP mechanism presumes that the true state is 1 if ˆ v 1 ˆ q 1 > ˆ v 0 ˆ q 0 ; otherwise, it is presumed that the true state is 0 . In general settings, the SP mechanism determines the true state as the signal s that maximizes ˆ v s ˆ q s .

To facilitate comparisons between alternatives, we define the SP-inspired score as

In the case of binary signals, we set φ ( 0 )=-1 and φ ( 1 )= 1 . This results in the SP-inspired score being S (( ˆ x i , ˆ p i ) i ∈[ n ])= ˆ v 1 ˆ q 1 -ˆ v 0 ˆ q 0 .

Difference between Surprisal-based Score and SP-inspired Score Recall our Empirical Surprisal-based Score is defined as

Both our Surprisal-based Score and the SP-inspired score involve reconstructing the prior from agents' predictions. However, they differ in how to exploit this reconstructed prior to obtain a calibrated score. Our Surprisal-based Score is invariant of noise, meaning it does not change in the presence of noise when the number of reviewers is infinite. On the other hand, the SP-inspired score is affected by noise. This implies that in certain situations, even when we have an infinite number of reviewers, the SP-inspired score fails to distinguish which of the two papers is better. However, our Surprisal-based Score exhibits lower stability compared to the SP-inspired score, because it involves a division of the correlation, which may lead to a high variance when the number of reviewers is small.

Numerical Experiments We perform numerical experiments to compare the performance of our Surprisal-based Score and the SP-inspired score in the binary setting ( Σ ={ 0 ( reject ) , 1 ( accept )} ). We employ the same experimental setup as described in Section 6. Figure 6 and Figure 7 present the experimental results. Upon analyzing the results, we observed that the performance of our Surprisalbased Score and the SP-inspired Score is roughly the same when the number of reviewers is small.

0.0

0.2

0.4

0.6

0.8

1.0

B

0.5

0.6

0.7

0.8

A

=0.0

0.0

0.2

0.4

0.6

0.8

1.0

B

0.5

0.6

0.7

0.8

A

=0.3

0.0

0.2

0.4

0.6

0.8

1.0

B

0.5

0.6

0.7

0.8

A

=0.6

Our Method (Beta(.5,.5))

SP (Beta(.5,.5))

Our Method (Beta(1,1))

SP (Beta(1,1))

Our Method (Beta(3,3))

SP (Beta(3,3))

Our Method (Beta(.5,.5))

SP (Beta(.5,.5))

Our Method (Beta(1,1))

SP (Beta(1,1))

Our Method (Beta(3,3))

SP (Beta(3,3))

(a)

3

reviewers

0.0

0.2

0.4

0.6

0.8

1.0

B

0.5

0.6

0.7

0.8

0.9

A

=0.0

0.0

0.2

0.4

0.6

0.8

1.0

B

0.5

0.6

0.7

0.8

0.9

A

=0.3

0.0

0.2

0.4

0.6

0.8

1.0

B

0.5

0.6

0.7

0.8

0.9

A

=0.6

Our Method (Beta(.5,.5))

SP (Beta(.5,.5))

Our Method (Beta(1,1))

SP (Beta(1,1))

Our Method (Beta(3,3))

SP (Beta(3,3))

Our Method (Beta(.5,.5))

SP (Beta(.5,.5))

Our Method (Beta(1,1))

SP (Beta(1,1))

Our Method (Beta(3,3))

SP (Beta(3,3))

(b)

5

reviewers

0.0

0.2

0.4

0.6

0.8

1.0

B

0.5

0.6

0.7

0.8

A

=0.0

0.0

0.2

0.4

0.6

0.8

1.0

B

0.5

0.6

0.7

0.8

A

=0.3

0.0

0.2

0.4

0.6

0.8

1.0

B

0.5

0.6

0.7

0.8

A

=0.6

Our Method (Beta(.5,.5))

SP (Beta(.5,.5))

Our Method (Beta(1,1))

SP (Beta(1,1))

Our Method (Beta(3,3))

SP (Beta(3,3))

Our Method (Beta(.5,.5))

SP (Beta(.5,.5))

Our Method (Beta(1,1))

SP (Beta(1,1))

Our Method (Beta(3,3))

SP (Beta(3,3))

(a)

3

reviewers

0.0

0.2

0.4

0.6

0.8

1.0

B

0.5

0.6

0.7

0.8

0.9

A

=0.0

0.0

0.2

0.4

0.6

0.8

1.0

B

0.5

0.6

0.7

0.8

0.9

A

=0.3

0.0

0.2

0.4

0.6

0.8

1.0

B

0.5

0.6

0.7

0.8

0.9

A

=0.6

Our Method (Beta(.5,.5))

SP (Beta(.5,.5))

Our Method (Beta(1,1))

SP (Beta(1,1))

Our Method (Beta(3,3))

SP (Beta(3,3))

Our Method (Beta(.5,.5))

SP (Beta(.5,.5))

Our Method (Beta(1,1))

SP (Beta(1,1))

Our Method (Beta(3,3))

SP (Beta(3,3))

(b)

5

reviewers

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles a problem in peer review where reviewers may provide noisy/biased ratings for papers. The paper investigate a \"one-shot\" scoring process inspired by the \"Surprisingly Popular\" method, that can rank papers by their true quality without any prior knowledge, even if different papers have different noise levels and biases.\n\nTheir method relies on eliciting reviewers’ predictions for a random reviewer’s signal and using this for calibration. Specifically, their model assumes that there is a prior distribution over the states (where each paper's state in the \"clean\" setting is the distribution of reviewers’ signals)  and that reviewers are perfect Bayesians who can update their beliefs based on their private signals. In the \"noisy\" setting, reviewers can only observe noisy versions of their signals, determined by a random mapping that depends on the noise level and the bias vector.\n\nThen, the paper defines the true quality of a paper as the expected score in the clean setting, and aims to design a method that can rank papers by their true quality with high probability, even if they have different noises/biases. For this, they introduce a surprisal vector that measures how \"surprising\" a paper’s state is compared to the prior distribution, and is normalized by the correlation between reviewers’ signals. The key result is that the surprisal vector is noise-invariant, i.e., the surprisal vector is the same in the clean and the noisy setting, regardless of the noise model.  Then, by computing an empirical surprise-based score for each paper, the mechanism papers can be compared in a noise-invariant way and are consistent with their true qualities.\n\nSTRENGTHS:\n- The paper is tackling an important problem, which is the question of how to de-bias and de-noise reviewers to obtain a clearer view of paper quality.\n\n- The observation that the surprisal vector is invariant under noise is insightful and interesting, as was the idea to look to the surprisingly popular mechanism.\n\nWEAKNESSES:\n- The presentation of the paper could be improved. The intro feels a bit unfocused, there are a number of grammatical errors, and the paper is a bit notation heavy. It might help to move the notation table from the appendix to the main body, and also expand on the \"high level ideas\" to more provide a clearer overview of the proposed method. It took a couple reads to understand exactly what was being proposed. \n\n- The assumption of calibrated agent predictions is not clearly justifiable in a context where agent signals are assumed to be potentially biased.\n\n- In line 162, the result cited by Kong et al. (2018) is outdated, suggest updating with more recent results that improve on this:\n\nSchoenebeck, Grant, and Fang-Yi Yu. \"Two strongly truthful mechanisms for three heterogeneous agents answering one question.\" ACM Transactions on Economics and Computation 10.4 (2023): 1-26.\n\nSrinivasan, Siddarth, and Jamie Morgenstern. \"Auctions and prediction markets for scientific peer review.\" arXiv preprint arXiv:2109.00923 (2021).\n\nQUESTIONS:\n- How do you handle det(U) being negative in practice?\n\n- Can you give some intuition for why is the normalization of the surprisal vector important?\n\n- Do the exponential bounds in the general case not depend on the cardinality of the signal space? Can you give some intuition for why this is the case? \n\n- Are the results robust if noise in agents' predictions of other agents' signals correlated with the noise in their own signals?\n\nLIMITATIONS:\nThe main limitation I see is that it is unclear what to do if some of the computed statistics (like det(U)) violate the non-negativity requirement.\n\nThe authors are up front about other limitations. These aren't ideal, but may be OK for the goals of this paper.\n- Authors assume no incentive issues, i.e., that all agents report truthfully. However, this is unlikely to be the case in practice, especially if participants know that there is \n\n- The authors assume that agents are calibrated in their predictions, even if their signals may be biased. This does feel like a bit of an odd assumption, and is unlikely to be true. Why would signals be biased, but predictions of others' signals be calibrated?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to calculate peer-reviews scores for papers in the presence of systematically biased noise, such that the score of a paper with a higher expected score in a noise-free regime is higher than the score of a paper with a lower expected score in a noise-free regime with high probability (as the number of peer reviewers grows large). The method does not use reviewers' historical scores to compute their priors. Instead, the method asks reviewers to predict the scores of other reviewers (similar to the Bayesian truth serum). Numerical experiments confirm the theoretical results that the proposed score is better able to distinguish between \"good\" and \"bad\" papers under systematically biased noise compared to a baseline that just averages the uncalibrated peer review scores.\n\n**Post-rebuttal:** I am raising my presentation score to 4 (excellent).\n\nSTRENGTHS:\nThe paper is clear and precise, and spends sufficient time setting up the problem and building intuition. The proposed method adapts a well-known idea (Bayesian truth serum) to the setting of systematically-biased noise in peer review in a principled manner; this is novel to the best of my knowledge. It is also non-trivial, and provides a useful way to construct peer-review scores without needing historical data for each peer reviewer.\n\nWEAKNESSES:\nThe generality of the exposition also makes it a bit difficult to follow. I think just limiting the main body of the paper to the binary decision case (accept, reject) and adjusting the notation for this (eg. a for accept, r for reject) would make it easier to follow the main argument of the paper. The generalization to multi-valued decisions (eg. review scores) could be relegated to an appendix or to just one theorem.\n\nThe numerical experiments do not consider SP as a baseline (possible with some modifications since it cannot be directly applied). A naive application of SP would be the most related method in prior work for the problem considered in this paper. Hence, it would help to see how this would perform.\n\nQUESTIONS:\nQ. Could the authors report numerical results from a naive application of SP as a baseline?\n\nLIMITATIONS:\nThe authors have noted the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of comparing two papers based on noisy ratings, where the noise can be arbitrarily biased for different papers. The paper elicits from each reviewer both a rating and a distribution of predicted ratings from other reviewers’ (based on a Bayesian update of the common prior, known to the reviewers). The paper proposes a scoring method that recovers the correct comparison between papers from the noisy reports, which essentially corrects the reported ratings using the reported prediction distributions. The authors theoretically show a bound on the error probability of a comparison based on their scoring method, and experimentally evaluate the comparison accuracy.\n\nSTRENGTHS:\n- The problem considered by the paper, calibrating biased noise in a one-shot comparison between items, is conceptually interesting. The authors motivate this problem well in the peer review setting with examples of cases where multiple reviewers may have the same bias. \n- The authors prove a theoretical bound on the convergence rate of their method’s error probability to 0. The experimental settings considered are thorough (and demonstrate aspects of the theoretical result).\n\nWEAKNESSES:\n- While the problem itself is interesting, the assumption that all agents share and know a common noisy prior is very strong; e.g., in the peer review examples in the introduction, while the noise may be biased in the same direction for all reviewers, the noisy prior may not even be similar for all reviewers. Given this assumption, I’m unsure about the significance of the proposed method, which essentially recovers this prior (as in past work) and uses it to standardize the reports.\n- I found the clarity of the writing to be generally poor, leading to some significant confusion at points (particularly in Sections 1 and 2). For example, in Section 1, I did not understand the claim following Example 2 (that a higher noise level would result in a lower expected score).  In Section 2, it was difficult to follow which aspects are observable by agents and by the mechanism.\n\nQUESTIONS:\n- Could the authors expand on what makes the setting of this work unique as compared to the settings of the past works referenced in Section 1.1 (Lines 123-129)? Are none of these works applicable as a baseline for comparison?\n- The invariant surprisal vector concept is given a lot of focus (Section 3), when the purpose of this concept seems to only be that it’s used to prove Corollary 1. Since the surprisal-based score (in Section 4) is the main contribution, I might suggest making it more clear in Section 3 what the purpose of the invariant surprisal vector is, in order to keep the focus on the main contributions. \n- Could the authors clarify the aspects of Section 1 and 2 referenced above?\n\nLIMITATIONS:\nThe authors adequately address the limitations of the assumptions made (perfect Bayesian and identical reviewers).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to detect and correct bias in Peer Review. They propose a one-shot noise calibration process without any prior information. Experiments are conducted on the binary case to show the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. The studied problem is important.\n2. Theoretical guarantee is provided for the proposed calibrated score.\n3. The organization of this paper is clear.\n\nWEAKNESSES:\n1. The rationality of the proposed surprisal scores needs further support. For example, in Figure 1, it seems that the left paper with lower negative reviewers’ prediction is the same as the right paper with higher negative reviewers’ prediction, while the left paper receives 1 accept and 2 reject while the right paper receives 2 accept and 1 reject. This result is confusing since the reviewers may give low P_0,1 simply due to the poor quality of the left paper.\n2. The experiments are weak in demonstrating the generality of the proposed method. For example, in Sec 6, the authors only conduct experiments on the binary case, lacking the more general settings with more types of possible signals. \n3. The theoretical guarantees lack a clear explanation. For example, in theorem 2, why there is 1/2 in the error probability (Pr[S(A) > S(B)|w1^A,w2^B]+1/2 Pr[S(A)=S(B)|w1^A，w1^B])? The guarantee of such an error probability may still remain a gap in the performance guarantee of the proposed method.\n4. Some assumption or claims lacks further support and needs to offer more clarification or explanation. For example, in Sec 2.2, why “only consider the noise where M is invertible?” What will be the situation when M is non-invertible? Why can construct the vector q and the joint distribution matrix U from the prediction matrix P as stated in claim 1?  Is it universally applicable to assume that 'each individual receives the clean signal with probability 1-\\lambda’?\n\nQUESTIONS:\n1. In theorem 2, why there is 1/2 in the error probability (Pr[S(A) > S(B)|w1^A,w2^B]+1/2 Pr[S(A)=S(B)|w1^A，w1^B])? The guarantee of such an error probability may still remain a gap in the performance guarantee of the proposed method.\n2. In Sec 2.2, why “only consider the noise where M is invertible”? What will be the situation when M is non-invertible? \n3. Why can construct the vector q and the joint distribution matrix U from the prediction matrix P as stated in claim 1?  \n4. Is it universally applicable to assume that 'each individual receives the clean signal with probability 1-\\lambda’?\n\nLIMITATIONS:\nThe authors have discussed the limitations of this paper in Sec 7.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tries to solve famous problem of removing bias and noise during peer-review process. The authors address the problem in one-shot setting (without historical data) and propose a novel approach that allows to remove influence of bias and noise (under some assumptions on bias and noise) on the ranking of assessed elements (reviewed papers). The approach uses scoring adjustment (used for ranking) and requires additional signal (action) from each reviewer. \nThe authors provide theoretical guarantees and small synthetic experiments.\n\nSTRENGTHS:\n-\tOverall good organization of the paper storyline (very good Intro!)\n-\tMore or less clear statements and easy to follow\n-\tSynthetic experiments\n\nWEAKNESSES:\n-\tSome drawbacks in clarity related to assumptions and limitations of the setup\n-\tSeem limited practical effect\n-\tThe contribution does not look very enough for NeurIPS\n-\tLack of comparison with alternative ways to improve scoring (e.g., non-1-shot ones, etc)\n\nQUESTIONS:\n1. It would be nice to understand whether the assumptions are realistic (for instance, Lines 59-60 “the correct answer is positively correlated to each agent’s signal” ; Lines 82 83 “assume that the noise is positively correlated .. ”; and the other). Right now the assumptions look theoretical with no clear understanding, e,g., how frequent they are in practice.\n\n2.  The specific assumption about particular cases of noises (Lines 203-205) looks very important while being deferred to Sec3 as Definition 1 (instead of being kept in Section 2 where environment of the problem is described). It makes feeling of overclaiming in Intro, where our Results (Lines 85-91) are stated without assumption of considering invertible noise (while other assumptions and limitations are addressed before Line 85). It would be nice to have some presentation improvements to address this gap. \n\n3. The authors claim that it is nice to have the result about best ranking when the number of reviewers goes to infinity. However, I believe it is not so important result to initially stated problem, where the limitations on the number of reviewers is crucial part of the problem (in most peer review practice we deal with 3-5 reviewers, e.g., see the case of NeurIPS). So, OK to have such theorem, but this result seems not so important and relevant to the problem taken for the research. It would be nice, if the authors focus at least the presentation more on how their results help in real practical cases like n = 3-5. (Yes, I see that the authors are aware of such numbers by providing experiments with such numbers. But (a) it might be just because of simplification of experimentation; and (b) the main contribution lies in the theoretical part of the work).\n\n4. While reading from the first page till the end, I had observed changing perception (for sure, mine, subjective) of the studied setting. From the beginning of the paper till Page 6 (Lines 189-190), I thought that the setup assumes that all reviewers assess all papers, while, in practice, each agent review few numbers of papers. Sections 2.1 2.2 have made me sure that the setup is so strange. Only, Lines 189+ reverted my perception. So, I would strongly recommend to somehow help such readers as I figure out this earlier than Page 6.\n\n5.The statement in Lines 168-169 «we focus on homogeneous noise setting where the noise is the same for all reviewers and will discuss the heterogeneous setting in Section 7\" makes feeling of covering “heterogeneous setting” by this work. After reaching Section 7, I have found that Section 7 is Conclusion section and heterogeneous setting are discussed as some future work... Missed expectation\n\n========\n\nAFTER REBUTTAL\nI thank the authors for answering the questions. I hope that the answers will be reflected and fully addressed in the new revision of the work.\n\nLIMITATIONS:\nSome presentation improvements may address limitations better\nSee points 1, 2 and 3 in the field “Questions”.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}