Least Square Calibration for Peer Reviews
Authors: Sijun Tan
Jibang Wu
Xiaohui Bei
Haifeng Xu
Venue: NeurIPS 2021 Poster (2021)
Keywords: peer review
calibration

Abstract: Peer review systems such as conference paper review often suffer from the issue of miscalibration. Previous works on peer review calibration usually only use the ordinal information or assume simplistic reviewer scoring functions such as linear functions. In practice, applications like academic conferences often rely on manual methods, such as open discussions, to mitigate miscalibration. It remains an important question to develop algorithms that can   handle different types of miscalibrations based on available prior knowledge. In this paper, we propose a flexible framework, namely \emph{least square calibration} (LSC), for selecting top candidates from peer ratings. Our framework provably performs perfect calibration from noiseless linear scoring functions under mild assumptions, yet also provides competitive calibration results when the scoring function is from broader classes beyond linear functions and with arbitrary noise. On our synthetic dataset, we empirically demonstrate that our algorithm consistently outperforms the baseline which select top papers based on the highest average ratings.


Least Square Calibration for Peer Reviews
Sijun Tan"
Jibang Wu" Department Computer Science University of Virginia Charlottesvilll, VA 22903 jw7jbevirginia. edu
Department of Computer Science University of Virginia Charlottesville, VA 22903 stäeuśvirginia.c edu
Xiaohui Bei
Haifeng Xu Department of Computer Science
School of Physical and Mathematical Sciences Nanyang Technological University
University of Virginia Charlottesville, VA 22903 hx4ad@virginia.. edu
Singapore 637371 xhbei @ntu. ent.iiuttuuuuuggg edu.
Abstract
Peer review systems such conference paper review often suffer from the issue miscalibration. Previous works on peer review calibration usually only use the ordinal information assume simplistic reviewer scoring functions such linear functions. In practice, applications like academic conferences often rely on manual methods, such as open discussions, to mitigate miscalibration. It remains an important question to develop algorithms that can handle different types of miscalibrations based available prior knowledge. In this paper, propose a flexible framework, namely least square calibration (LSC), for selecting top candidates from peer ratings. Our framework provably performs perfect calibration from noiseless linear scoring functions under mild assumptions, yet also provides competitive calibration results when the scoring function is from broader classes beyond linear functions and with arbitrary noise. On our synthetic dataset, we empirically demonstrate that our algorithm consistently outperforms the baseline which select top papers based on the highest average ratings.
Introduction
Peer review has been essential to the advancement of academic research. In typical conference peer review process, the submitted papers are assigned to a set of external reviewers and will receive cardinal scores from them. The final decision of which papers get accepted to the conference is usually determined by the average score received for each paper. However, during such process, different reviewers may have different standards and biases in their evaluations. For example, the same paper may receive high score (c.g., 8 out of 10) from a lenient reviewer and a low score (e.g.. 2 out of 10) from some strict reviewer. As result, the average score may always reflect the true quality of paper. This issue is known as miscalibration, and is prevalent problem in many other online review systems well such TripAdvisor Yelp. In order to ensure fair and unbiased outcome, it crucial to the concept and to the success of a peer reviewing system to have an efficient and effective calibration algorithm.
There has been body of works attempting to address the miscalibration issue in peer reviews. Most of these works can be categorized into two directions. The first line of works relies making simplifying assumptions on the form of miscalibration. For example, [9 10 16 27
Equal Contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

model the miscalibration as one one-parameter additive bias for each reviewer; [19 21 assume each reviewer's scoring function to be linear function. Many of these works are able to accurately learn the unknown parameters of the miscalibration due to its simple structure. However, these strong structural assumptions on the miscalibration are often too strong for these results to widely applicable. The second line of research [20, 9 13 18
K allows the miscalibration to be arbitrary, and only relies on ordinal ranking information obtained from cardinal scores to calibrate. A recent work 26 bridges these two lines of research by showing that even when the reviewers' functions are arbitrary, there exist cardinal score estimators that perform better than any ordinal ranking estimators. However, their work is still pertained to worst case assumptions, and is restricted to the scenario where each reviewer only reviews a pair of items. It remains an important open problem to design calibration models and algorithms that can work with weaker simplifying assumptions (e.g non-linear functions) in a practical scenario.
Summary of Contributions. In this work, we propose new optimization-driven unsupervised learning framework, least square calibration (LSC). As the name suggests, the framework leverages the natural and powerful least square method for handling different types of miscalibrations in peer reviews. A key strength of our framework is its flexibility in incorporating various levels of prior knowledge, which consequently results in different qualities of calibration. Specifically, we capture the system's 's prior knowledge about reviewers' scoring as a hypothesis class of scoring functions. Depending on how much the system knows about reviewers, the hypothesis class can be "rough" as all monotone functions, or as specific as linear functions, convex/concave functions, Lipschitz continuous functions, even their mixtures. Our framework can be tailored to accommodate all these hypothesis classes and moreover, cope with the reviewers' noisy perception of true item qualities.
We demonstrate the effectiveness of the LSC framework both theoretically and empirically. We start our theoretical analysis from the hypothesis class of linear scoring functions. When reviewers' scores are generated by arbitrary linear functions of true item qualities without noise, our main result is a necessary and sufficient characterization about when LSC can (essentially) perfectly recover true item qualities. Interestingly, it turns out that perfect recovery is fully captured by a novel graph connectivity notion for a carefully constructed reviewer graph, which we coin double connectiviry. This result reveals useful insights about what kind of reviewer-item assignments may be good for calibration, which is further justified in our experiments. When reviewers perceive item qualities with noises, we show how reduce the LSC framework scoring functions are linear. We then extend this technique to accommodate other function classes including monotone, convex, or Lipschitz continuous functions. Our theoretical analysis is concluded with discussion about the connection of LSC to other well-known problems such as regression and matrix seriation.
tractable convex optimization program when
Empirically, we test the performance of LSC via extensive experiments. We show that LSC consis- tently outperforms previous methods in various problem setups. The experiments also demonstrate that the calibration quality of LSC improves as our prior knowledge improves and LSC is robust to reasonable amount of misspecified prior knowledge. We hope these encouraging findings can motivate more future studies about how to identify more accurate class of reviewers' scoring functions.
2 Problem Formulation
In this section, will first formally describe the peer review problem and then introduce calibration approach. We use conference peer review as an illustrative example, though our model and methods can be trivially generalized many other peer review systems.
The Peer Review Problem. In a peer review process such as an academic conference, there are N items and M reviewers. By convention, we use [N] = {1,-.. N} to denote the set of items and [M] = {1, ... , } to denote the set of reviewers. We assumes that each item has an inherent ground truth quality x= (z). Throughout the paper, the superscript notation "x" will always be used to denote true qualities. For simplicity, we assume that there are no ties and the item qualities are all distinct from each other. Each reviewer j reviews subset of the items I, S [N]. For now, we allow I, to be an arbitrary subset, which can even be a singleton or the entire set of items [N]. We will specify later under what conditions about (T, beM our algorithms will work. We denote 13 E [N] as the index of the f'th item reviewed by r.viewer
2

---

Scoring Functions and Score Generations. Each reviewer j E [M] is characterized by monotone
scoring function fj : I + R. Like most previous works in this space Ta 21 16 27 we assume that reviewers' scores are generated by applying her scoring function to noisy perception of the item's true quality. Formally, reviewer j's score about item 1j. denoted by vf. is generated as follows
Score Generation: vj -= fy(2" (T)) +5)
(1)
where /j is reviewer j's scoring function and ef is an independent zero-mean noise. For convenience, we always assume the scores for each reviewer j is ordered increasingly such that v) < 1) yj 1
Design Goal. We are interested in designing an item selection algorithm. The algorithm takes as input the item assignment [TN] for a set of items [M] and set of reviewers [M], the reviewers' scores for each of their assigned item {v5) s The goal of the algorithm is to output a set of n items that best matches the top n items according to the (unknown) true qualities x² In the experimental section, we will describe multiple metrics to evaluate the performance of our algorithm.
and additional threshold parameter I N.
Restricted Classes of Scoring Functions. Note that when defining reviewers' scoring functions, we do not make any assumptions on the properties of these functions other than monotonicity. This is to ensure that our framework is flexible and can handle wide range of problems.
However, in many situations, the designer may have some additional prior knowledge and infer more structures about the scoring functions. The reason of assuming some restricted class of scoring functions is multifold. First, such functions may be good proxy or approximation of the true scoring functions based on our prior knowledge. Second, it may help us to avoid the issue of overfitting since allowing arbitrary scoring functions may be able to fit the review scores well but leads to poor generalization and make the result prone to perception noise. For example, many previous works 21 10 5 16 have focused on the basic model of linear scoring functions, defined follows.
Definition 2.1 (Linear Scoring Function). A linear scoring function of reviewer j has two parameters: az 2 and by E TR. and of the form fj (z(= x + c)) = a, bj-
In addition to linear scoring functions, one may also consider broader class of scoring functions when less prior knowledge is available when we want to make less stringent assumptions. For example, our framework can also handle scoring functions that are convex or concave or mixture these functions f However, scoring functions are more realistic since this answer would depend on the concrete domains of the problem. Instead, our goal is develop a generic learning framework that can integrate any of such prior knowledge to improve the calibration. That being said, we believe it is an important future research direction to design heuristics, regularizations and other methods choose the hypothesis class for better calibration results as well as to understand the potential issue of overfitting for hypothesis class that are broad.
emphasize that it is not our purpose to justify which of these classes of
3 The OptimizationnDriven Framework
Central to our approach a framework to recover the underlying item quality x that can properly "explain" the observed review scores given the assumptions about the scoring function structures. Note that our recovered qualities may be up to some constant shifting or re- re-scaling from the true qualities. However, as long as the shifting and re-scaling preserve monotonicity, this will not affect the accuracy of our selection of the best n items.
Following the notation in machine learning, let H denote a hypothesis class of scoring functions. For example, for linear scoring functions, the hypothesis class is H = (f f() = ar + for some a 2 0, E JR.. We denote this linear scoring function class as Hr, for convenience. More generally, H may be any class functions depending on designer's prior knowledge of these scoring functions.
Given a hypothesis class H. our goal is to find scoring functions f1..... sas E H for all reviewers, such that the distance between the true qualities x and qualities recovered by these selected scoring
2 Overfitting here is in the similar n samples. or in low-rank matrix completion where is the target rank, n the matrix size.
degenerated cases such as setting k = in k-means clustering of
"A scoring function f' convex if for any æ1, X2 E R and for any & (0, 1), we have f(tx: + (1 - t)x2)) S
(211 (1 (x2) Function is concave if -f is convex.
3

---

functions is minimized. In this paper, we use the Euclidean distance. This leads to the following general optimization-basee based framework, which we term Least Square Calibration (LSC). Note that in general LSSC is functional optimization problem (FOP) with: (1) functional variables f = {J} H) He)E (2) item quality variables x = (=(1)) (3) noise variables = = (<)))E(M,,eIII,Il
Least Square Calibration (LSC)
M 11 ΣΣSS j=1
min x,f.c
(2)
s.t. (x(I)) +())
vj [M] e [4]
where the set cardinality 1 is the number of items reviewer j reviews. LSC tries to find the scoring functions and true qualities that best fit all the review scores terms of the least square error.
The rationale behind LSC can be seen from several perspectives. First, fix any (unknown) parame- ters x and f. define random variable Y - ʃu (z(1 + with randomness inherited from the noise ef. Consider the following parameter estimation problem given observed realizations of Y as vf. find parameters and f to maximize the likelihood of the observation. It is not difficult to see that the optimal solution to LSC is the maximum likelihood estimator, when cf is i.i.d. Gaussian noise with mean O. Second, it turns out that LSC can also be viewed as variant of the classic matrix seriation problem. In Section we will discuss this connection in more details as well as the connection of LSC to linear regression.
3.1 Linear Scoring Funetions with No Noise
LSC is a general calibration framework for recovering the true qualities of the items. One cannot hope to solve it efficiently without any assumption on the scoring function hypothesis class, since functional optimization problems are generally intractable. Fortunately, we will show that for broad class of commonly used hypothesis classes, LSC can be formulated as convex programs and therefore be solved efficiently. In some situations, we even show that LSC can provably recover the correct item qualities perfectly under mild assumptions.
As warm-up, we first restrict our attention to perhaps the mostly widely studied hypothesis class, i.e., linear functions. We start with the simple situation with no noise. this case, it not difficult to show that the following linear feasibility problem solves the LSC Problem (2). We omit the proof of this claim since a more general result will be proved in subsection
LSC with linear scoring functions and no noise
min x
0
(3)
s.t.
x x(1)) x1(1--11 ) 21
vj E [M] 42 < E |
a(1)) x x(15-1)
x(
-
Vj E [M] 2 < f < [7, --
With the prior knowledge that the scoring functions are all linear functions and that there no noise, it is easy to see that the true item qualities x" and scoring functions f" must be feasible solution to LP B However, we cannot guarantee that LP ( doess not have any other feasible solutions whose induced item order disagrees with x". This could happen, for example, when the reviewers do not review enough items, or their assigned items do not have enough overlap to perform a full calibration. Therefore, it is important question to understand under what condition we can (essentially) recover all item qualities using LP in the following sense:
4 'The parameters in this estimation problem includes functional parameters. 'Here to guarantee x(15) to be strictly larger than x<15-1) and we w.l.o.g. use 2 1 for convenience since rescaling the solution will not change ranking.

---

Definition 3.1 (Perfect Recovery). A set of qualities {z(1) is said to perfectly recover the true qualities {a" (E(N if there exists >> 0 and such that z²i) = or (i) + foror every i E [M.
We remark that perfect recovery is stronger condition than recovering just the correct order of the items. It additionally requires that the quality gaps among different items are proportional to the gaps of true qualities. One might wonder whether such strong goal of perfect recovery may be too good to be possible. Surprisingly, our next result formally show that if the reviewer assignments satisfy some reasonable conditions, it indeed possible to hope for perfect recovery. To introduce our result, we introduce novel notion of a review graph and the property of recover-resilience.
Definition 3.2 (Review Graph and Recovery- Recovery-Resilience) A review graph is an undirected multi- graph G = (V. E) where V is the set of reviewers, and there are I[ n I edges connecting every two reviewers i. E V. A review graph G is said to be recovery resilieets if for any item assignments inducing G and any given review scores {u] j.s, any solution to LP 3 a perfect recovery.
That is, the review graph is graph that connects reviewers to each other according the number of common items that they reviewed. turns out that the recovery- recovery-resilience resiliecee of review graph is fully characterized by novel notion of graph connectivity, which we coin the double connectivity.
Double connectivity naturally generalizes the standard notion of graph connectivity. Recall that a connected component of the reviewer graph is simply subset of reviewers $ S [M] who connects to each other. We will refer to the items they have collectively reviewed, C(S) = Uies
s Is, as the items covered by component S. A connected component of graph can be identified through a simple on -niin joining two components whenever they share at least one edge. Our notion of doubly-comnected component include all components that can be found using the same joining procedure, except that each time we join two components only when they share at least two edges. A formal procedure, coined SeppattUniooes is given in Algorithm T below.
Repeat-Und
procedure, where we start by treating each node as a component and then repeatedly
Algorithm 1 RRReeat-
input Review graph with N item, M reviewer, and item assignment set (K)}eEM
1: U <- [1]: [2]....: [M])
2: while 35,, S2 E U. s.t. (((S))C(SS)) 2 2 do
3: U S2 + (S1 U S2) (i.e., merge St. S2)
4: end while
5: return U
Definition 3.3 (Doubly-connectee component). Any component that can be generated by Algorithm is called a doubly-connect component.
The following theorem shows that to guarantee perfectly recovered qualities from LP 3 under linear scoring functions without noise, all we need is doubly-coomected component that covers all items.
Theorem 3.4. A review graph G is recovery-resilient if and only if the review graph G has doubly-comnected component S that covers all items, i.e. C(S) = [N].
Proof Sketch. The proof is somewhat involved particularly for the necessity of double-connectiiity. We defer the full proof to Appendix and only describe the proof sketch here. For the "if" direction, the key idea is to use scores of at least two shared items between two reviewers ji and ja calibrate the other items they reviewed in 1,, 4iz- After this calibration, we can effectively treat these two reviewers as single "reviewer" (formally, a doubly-connected component) who reviews all the items 1, U Liz- Algorithm follows exactly this inductive procedure by repeatedly merging two components into a single component, until there a "reviewer" that reviewed the entire item set [N], and these are the calibrated qualities that we want.
The "only if" direction is more challenging. Here, we have to prove that for any reviewer graph that does not have a doubll-connected component covering N, there must exist an instance where perfect recovery is impossible. The main difficulty lies crafting such an instance, by specifying paper assignment, true item qualities etc., for any such graph. Our proof leverages nice properties of prime
AA multi-graph is graph that allows multiple edges between two vertices.
5

---

numbers to carefully construct the instance such that it will provably lead to at least two linearly independent solutions of LP
Remark 3.3. Theorem 3.4 provides useful and practical insights for reviewer assignments in real applications. Specifically, it appears that stronger connectiviry among reviewers during review assignment, defined reviews. In practice, such connectivity can be achieved as follows: (1) run Algorithm 7 (2) if the algorithm stops without a doubly connected component covering all items, then simply ask any reviewers in one component to generate two reviews for two additional items covered by the other component so that these two components now become suuble-connectd. This procedure is actually way for calibration during post-rebuttal
fermmaly formally in the sense of Definition 3.3 could be more helpful to calibrate their
common during today's 's academic conference review discussions. Interestingly, Theorem F.4 serves as a theoretic justification for this practice. In Appendix B.2 provide experiment results which further suggest that the topological structure of double connectivity exhibits stronger robustness against perception noise.
3.2 Linear Scoring Functions with Noise
The noiscless case discussed in the previous section has an important feature: each reviewer's s reviewing scores are monotone in the items' true qualities. This a very useful property that could help the design of efficient calibration algorithms. However, it is also restricted assumption that is often violated in practice. In real-world conference reviews, it is very common to see two reviewers reviewing the same pair of items but rating them in different orders. This is typically explained by reviewers' noisy perception of the paper qualities perception noise in score generation procedure
[T0 21 5
27 This leads to our model of 16
LSC with linear scoring functions and with noise
min X,
Σ Σ (c))
(4)
1 Vj
s.t.
1-1
VjE [M,,25 [4]
C
7
Jf41 T..
Vj E [M], 2 < 141-
y
x of
VjE M1 1 < e f 4
As discussed before, assume each ef is independent noise added to the true quality of the item.
Let (C) denote the set of all increasing linear scoring functions whose derivatives (whenever exists) are upper bounded by some constant C. More formally, C' = supf æ:-z; practice, C' can be empirically chosen to be large constant if prior knowledge is lacking.) After adding the noise to the linear constraints in LP B our LCS with linear scoring functions becomes the above convex program, whose validity is guaranteed by the following theorem.
f(=:)-((a)) (Vz,, æz}. (In
Theorem 3.6. Convex Program is equivalent to LSC with H (C) in the following sense: for any optimal solution (x* e*) to Program there exists f = Uf E Hc(C)) such that (x*, f*) optimal to LSC Program D
Theorem 3.6 shows that we can recover the same item order by solving from LSC or by solving CP 41
The proof of Theorem relies on a general procedure to argue the equivalence between a functional optimization problem and a variable optimization problem. We defer the formal arguments to Appendix A.2
3.3 Handling Other Classes of Scoring Functions
It turns out that the above approach in Section 3.2 can be generalized to handle prior knowledge on various types of function shapes such as linearity, concavity and convexity, reference information (c.g., knowing the true quality of a few items), derivative information such as Lipschitz continuity, etc.

---

For example, suppose all we know about the scoring functions is that they are monotone increasing, then Program can be adapted this case simply by removing its second linearity constraint
and only using the ordinal constraint z - 2 レうーي with a sufficiently large constant C. As another example, if reviewers known to have convex scoring functions, we can change the
linearity constraint the convexity constraint
These can all lead to similar
guarantees as Theorem 3.6
Additionally, another advantage of our optimization LSC framework is that it can handle a mixed of reviewer scoring functions, if such more refined prior knowledge is available. Specifically, in many applications, the algorithm designer may have different amount of prior knowledge about different reviewers since some (senior) reviewers have been in the system for a long time whereas some just entered the system. Consequently, the designer may know, e.g., some reviewers' scoring functions are linear, some are convex or concave, whereas for some other reviewers, the designer knows nothing beyond monotonicity. Our LSC framework can be easily adapted to handle such mixed prior knowledge since constraints can be "personalized" for each reviewer. Certainly, more specific or detailed prior knowledge about the hypothesis class will lead to better calibration, as also shown in our experiments. However, we remark that what hypothesis class is more realistic or justifiable will likely depend on concrete applications on hand, and is beyond the scope of this paper our focus here is on the generic methodologies. We will demonstrate this kind modeling strength of the LSC framework in the experiments section.
Connection to Other Problems
Our LSC framework shares connections to several other problems in statistics and machine learning. In the hope of providing further intuition and justification for our approach, in this section we elaborate on these connections.
4.1 Linear Regression
Linear regression 2 is classic approach to modeling the relationship between two more variables as a linear function. Ordinary least square (OLS) is a method to fit such a linear model by minimizing the sum of squared difference between the observed dependent variable and the predicted dependent variable. More specifically, OLS can be expressed by the formulation in Program 5
M
min a.B.c
(5)
min 3.0.3.c
(6)
s.t. = + 1I
s.t. v/ ay (x(1)) + (g))))+))+++))) +3, Vj [M] 1 (4g
Linear regression formulation
LSC formulation
Observe that the formulation of linear regression is structurally similar to our LSC when H = H L. which is formulated in Program (6) above. There are two major differences between our LSC in Program and OLS in Program 5 First, the true item qualities x in OLS (5) is known, whereas in our LSC x is unknown and treated as variables to optimize. Second, with the additional challenge of not knowing x, the LSC also has an advantage - we have multiple reviewers (indexed by j) with multiple linear functions/models who will generate scores for the same x. These make LSC relevant to but quite different from OLS linear regression.
4.2 Matrix Seriation Problem
Our framework is also closely relevant to well-known difficult combinatorial optimization problem called matrix seriation 22 [S]. The problem looks find a consistent ordering of the columns of a matrix. More specifically, given a set of objects and a (partially observed) data matrix A in which each row of A reflects partial observations of these objects, the goal of seriation to find linear ordering of these n objects that best fit the observed data according to certain loss function.
7

---

The matrix seriation problem has its roots in many disciplines, with applications such as sparse matrix reordering 4. DNA sequencing (6,, and archeological dating 77] There are many variants of the matrix seriation problem. Below we define a popular version with the f2 loss function.
Definition 4.1 (f2-Matrix Seriation). Given a partially observed matrix A E RTxxn in which each
entry Asg E R Ag = (not observed), find matrix B E mR that minimizes ||4 - BIβ
Lo.y (Aig Bıy)² subject Vi.j E [mm],q; 9.P p [m]: Bi.g S Big b B3,g S B3r-- That
is, in matrix B, the ordering of each entry according to the column consistent across each row.
Review score calibration can be naturally modeled as matrix seriation problem. We have a set of items, and data matrix A in which each row represents a reviewer's 's rating scores to (only) his/her assigned items. Our goal is to find an ordering of the items that most consistent with the scoring matrix A under the assumption that each review has a monotone scoring function. In the following. we show that the seriation problem is a small variant of our LSC problem.
Theorem 4.2. The l2 Matrix Seriation 44.1) problem can be solved by the following Functional Optimization Problem (Hmono contains all monotone increasing functions):
M 17
ΣΣ
min x.
|= S.L. w (x((() + e,J and ʃg E Hmmono
Vj E [M], S [4,]
is worthwhile to compare FOP 7 with our LSC 2 with monotone scoring functions as they have very similar formulations. The difference is that in FOP 77. the noise term eÿ is added after
applying the scoring function, whereas in LSC 2 it is added xxLt)) inside the function. Notably, this seemingly small difference turns out to greatly affect the tractability of the problem. Indeed, the matrix seriation problem is known to be computationally intractable [8] contrast, our LSC e
with monotone scoring functions can be efficiently solved via convex programming due to the noise
inside scoring functions (as assumed by most previous works in this space [IC 21
5 Experiments
A major challenge when evaluating the performance of calibration algorithms is that their performance measures rely the underlying true item qualities. Unfortunately this information is unattainable in most applications - indeed, we already know the true qualities, peer reviews would not be needed any more. Besides, due to anonymity of reviewers in all public peer review data that we are able to find, the underlying network of review graph cannot be recovered. Therefore, like many other works evaluate our algorithms primarily based on synthetic data where we do know the true qualities. To be more realistic. the distribution parameters of our synthetic data are chosen based on ICLR 2019 review scores M. Our experiments are designed according the review procedure of academic papers, we thus also refer to items as papers in this section. In addition, we include an experiment on real-world dataset 22 in Section 5.2 While our model still consistently demonstrates good performance, we still want to raise caveat here that strategic manipulation could be a negligible factor in the process of peer grading 29 24 Given that such strategic behavior is rarely observed in peer review setting such as academic conferences, it therefore remains a crucial task to obtain a real-world dataset for future work of this domain.
in this domain 16 26 277
5.1 Evaluation on Synthesized Dataset
Dataset Generation Each paper's true quality is drawn from Gaussian distribution (5.332, 1.2) truncated to be within [0, 10 I where the mean 5.32 and standard deviation 1.2 is estimated based on the distribution of ICRL2019 review scores. Our random assignment algorithm ensures that most papers will get reviewed by roughly the same number of reviewers. Each reviewer is randomly assigned a scoring function from pre-specified family. Before applying the scoring function, a zero-mean i.i.d. Gaussian noise ~ (O, σ) is added to the true quality of each paper for each reviewer. More details about the scoring function generate as well as the assignment algorithm can be found in Appendix B
'Truncation is for comparison convenience. Untruncated scores will only be easier since boundary papers are easier to distinguish.
8

---

Table I: Performance comparison under linear scoring setting (LSC is method). The left and right side of the table respectively corresponds to the noiseless (a = 0) and noisy (a = 0.5) setting.
Metric
Pre. (%)
Avg. Gap
Pre. (%)
Avg. Gap
Model
Average
40.0 + 4.0
0.78 + 0.08
39.2
4.5
0.80 + 0.08
QP
97.1 + 1.7
0.01 + 0.01
69.2 + 4.6
0.24 + 0.09
Bayesian
76.2 - 4.0
0.12
71.5 3.1
0.17 + 0.03
LSC (mono)
91.7 - 1.8
0.001
75.9 - 2.6
0.12 + 0.02
LSC (linear)
100
0
80.1
2.9
0.08 + 0.01
Evaluation Metries We adopt two different metrics to quantitively and comprehensively evaluate the performance of our models. Let T [N] be the ranked list of n papers with highest true qualities and S C [N] denote the ranked list of n papers selected by any algorithm. In all our experiments, we set SS] = [T| = 0.1N, i.e., an acceptance ratio 10% our simulated conference setting.
Precision P(S,T) 1 Eies 1/1 E T] measures the ratio of the top papers in T that are selected into set S. The higher precision is, the more papers with truly high qualities are accepted.
Average Gap a (S, T) Іo EieT (V) [st Eies x= measures the gap between the average true qualities of the best papers in T and average true qualities the papers in S. It characterize how far the recovered average paper quality is from the optimal. The smaller this gap is, the better.
1
Baselines We consider three competitive baselines. The first is a widely used heuristics which simply uses the averaged reviewers' scores prediction of the paper's true quality, and accordingly select the best papers. We will refer this model as the Average. The second baseline is based on quadratic program (QP) proposed by Roos et al. [1 The third baseline proposed by Ge et al. [0 uses Bayesian model (Bayesian) to calibrate paper scores. The latter two baselines both assume linear reviewer scoring functions with noise that is similar to our modeling assumption.
Experiment Setup We first test the situation with linear scoring functions in order to have fair comparisons with previous methods which mostly assume linearity. Parameters are set as N = 1000, .M = 1000, A: = 5. We compare two settings: (1) a 0 (noiseless case); (2) a = 0.5 (noisy case). All reported results are averaged over 20 trials. All of our models are implemented with the Gurobi Optimizer [ All our algorithms can be solved very efficiently in seconds, we thus will not present running time results.
Results Table presents the experiment results that compare methods with the three baselines. Notably, our model consistently outperform all baselines under both metrics in both settings. In the noiseless setting with linear scoring functions, the experiment results confirm analysis, as we indeed observe the perfect recovery by our model under this typical conference setup. In addition, while both LSC (linear), QP and Bayesian specifically models the prior knowledge of linear scoring functions, our model demonstrates the more robust performance regardless of the noisy environment in both metrics. This suggests that our model meets the objectives that want to accept the best papers and maximize the overall qualities of the selected papers. We also investigate how the model performance changes according to the hardness of the problem instance, such as the effect of changing number of papers per reviewer k, the paper to reviewer ratio : M, and the noise scale in our empirical study and plot the results in figures. In all experiments, LSC still consistently outperforms baselines, and more details can be found in the Appendix B
theoretical
5.2 Evaluations on the Peer-Grading Dataset
To test model beyond synthesized dataset, we well-known public Peer-Grading dataset from 22 that collects the peer grading scores of questions in 6 different homework submissions in an Algorithm & Data Structures class at the University of Hamburg. We use the average score graded by the TAs as the ground truth quality x of each paper (homework submission). In Table 2 we list
'The source code be found at https: httts:://gitai http::///tthbb www.//oommaaeeeeeee sigma/ssc

---

Table Performance comparisons in Peer-Grading Dataset.
Model
Average
QP
Bayesian
LSC (mono)
LSC (linear)
Metric
Pre. (%)
80.9
80.3
79,4
78.5
82.2
Avg. Gap
0.48
0.40
0.47
0.55
0.38
the precision of the LSC models and other baseline on selecting the top 50% of the first homework submission. Our model has the best calibration result. Moreover, the LSC model shows an even more clear edge ranking-basee metrics, meaning order the recovered quality matches better with the ground-truth. We defer the details of these results to Appendix B.6
5.3 Mixed Set of Scoring Functions and Robustness Mis-Specified Prior Knowledge
To study the robustness of the different methods in sce- narios where the scoring function assumption no longer holds is mis-specified, percentage of reviewers' scoring functions are instead ar- bitrary random (non-linear) monotone functions. Figure plots the precision curve of our methods and baselines with varying percentage of linear scoring functions. It shows that 1) our LSC (linear) consistently outperforms prior work (QP, Bayesian) that also assumes linear scoring func- tions. 2) As the percentage of linear function decreases, the gap between LSC (linear) and LSC (mono) becomes closer, and when all functions are random monotone func- tions, LSC (linear) is 4% behind LSC (mono) due to its mis-specified prior knowledge in the noiseless setting. 3) LSC (mix) which has accurate prior knowledge about which reviewers are linear- significantly outperforms LSC (lincar) and all other models, showing that good prior knowledge is indeed quite helpful for calibration.
consider settings where some
0.6
Bayadan
Average
0.0
100%
% of linear function function
Figure 1: Performance comparisons in the noiseless and mixed setups with lin- ear scoring functions and arbitrary mono- tone functions. Only LSC (mix) has prior knowledge of every reviewer's 's scor- ing function type.
Appendix B.4 contains additional experiments for noisy scores and mixed settings with monotone, convex and concave functions. The results further demonstrate the usefulness of prior knowledge for calibration and thus also justified the value of having a flexible calibration methods like LSC.
Conclusion
This paper presents a simple yet powerful framework for calibration peer review systems, which exploits both the robustness of linear regression methods and the topological structure of review graphs. Moreover, our empirical and theoretical results provide general guideline on the assignment rules in peer review for more effective calibration. In follow-up work, hope to build upon flexible framework for the extension to wider spectrum of scoring function hypothesis classes, as well as broader family of peer review systems beyond the classical format of academic conference. For example, AAAI-2021 recently started to adopt a review process senior reviewers review all the papers and make summary rejections. The remaining papers will go to the second phase and receive additional reviews for the final decisions. It remains interesting open problems develop novel calibration methods for such two-phase peer review process, or alternatively, to design the peer review mechanism to work better with calibration methods.
Pl where in the first phase
10

---

7 Acknowledgements
We thank all the anonymous reviewers for their helpful comments, especially the discussions on real-world datasets. This research is supported by Google Faculty Research Award and the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 (M002019-T2-1-045)
References
[1] 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. OpenReview.net. URL https:://openzeevirr net/ ec/2019/Conferenc
wwwp:/wwww.w...oo.doooo.. ggw.goodouppoooo ICLR
[2] The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty- Thirtt-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI
Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY.
USA, February 7-12, 2020, 2020. AAAI Press. ISBN 978-1-57735-823-77 URL https
//www. aaai wwwo/ogg/////.e htt//Lirrrrryrr http://AAAAaaaaaaootataaaoaa php
[3] Ammar Ammar and Devavrat Shah. Efficient rank aggregation using partial data. ACM SIGMETRICS Performance Evaluation Review, 40(1):355 2012.
[4] Jonathan Atkins, Erik G Boman, and Bruce Hendrickson. spectral algorithm for seriation and the consecutive ones problem. SIAM Journal on Computing, 28(1):297-310, 1998.
[5] Yukino Baba and Hisashi Kashima. Statistical quality estimation for general crowdsoureing tasks. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 554-562, 2013.
[6] Seymour Benzer. The fine structure of the gene. Scientific American, 206(1):70-87, 1962.
[7] George W Brainerd. The place of chronological ordering in archaeological analysis. In Americanist Culture History, pages 301-313. Springer, 1997.
[8] Victor Chepoi, Bernard Fichet, and Morgan Seston. Seriation in the presence of errors: Np- hardness of I-infinity fitting robinson structures to dissimilarity matrices. Journal of classifica- tion, 26(3):279-22) 296799999662999 2009.
[9] Yoav Freund, Raj Iyer, Robert E Schapire, and Yoram Singer. An efficient boosting algorithm for combining preferences. Journal of machine learning research, 4(Nov):933-969, 2003.
[10] Hong Ge, Max Welling, and Zoubin Ghahramani. A bayesian model for calibrating reviewer scores.
[11] LLC Gurobi Optimization. Gurobi optimizer reference manual, 2021. URL http: http::/wwwt www ourobi.co com
[12) Michael Hahsler, Kurt Hornik, and Christian Buchta. Getting things in order: an introduction to the package seriation. Journal of Statistical Software, 25(3):1-34, 2008.
[13] Anne-Wil Harzing, Joyce Baldueza, Wilhelm
Cordula Barzantny, Anne Canabal, Anabella Davila, Alvaro Espejo, Rita Ferreira, Axele Giroud, Kathrin Koester, et al. Rating versus ranking: What is the best way to reduce response and language bias in cross- national research? International Business Review, 18(4):417-44 432, 2009.
[14] Aggelos Kiayias, Markulf Kohlweiss, Petros Wallden, and Vassilis Zikas. Public-Key Cryptography-PKC 2020. Springer, 2020.
[15] Innar Liiv. Seriation and matrix reordering methods: An historical overview. Statistical Analysis and Data Mining: The ASA Data Science Journal, 3(2):70-91, 2010.
[16] Robert S MacKay, Ralph Kenna, Robert Low, and Sarah Parker. Calibration with confidence: principled method for panel assessment. Royal Society open science, 4(2):160760, 2017.
11

---

[17] Ioannis Mitliagkas, Aditya Gopalan, Constantine Caramanis, and Sriram Vishwanath. User rankings from comparisons: Learning permutations in high dimensions. In 2011 49th Annual Allerton Conference Communication, Control, and Computing (Allerton), pages 1 1143-1150 1150. IEEE, 2011.
[18] Sahand Negahban, Sewoong Oh, and Devavrat Shah. Iterative ranking from pair-wise compar- isons. Advances in neural information processing systems, 25:2474-2482 2012.
[19] SR Paul. Bayesian methods for calibration of examiners. British Journal of Mathematical and Statistical Psychology, 34(2):213-223, 1981.
[20] Milton Rokeach. The role of values in public opinion research. Public Opinion Quarterly, 32 (4):547-559, 1968.
[21] Magnus Roos, Jörg Rothe, and Björn Scheuermann. How to calibrate the scores of biased reviewers by quadratic programming. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, 2011.
[22] Mehdi SM Sajjadi, Morteza Alamgir, and Ulrike von Luxburg. Peer grading in a course on algorithms and data structures: Machine learning algorithms do not improve over simple baselines. In Proceedings of the third (2016) ACM conference on Learning@ Scale, pages 369-378, 2016.
[23] George AF Seber and Alan J Lee. Linear regression analysis, volume 329, John Wiley & Sons, 2012.
[24] Ivan Stelmakh, Nihar B Shah, and Aarti Singh. Catch me if i can: Detecting strategic behaviour peer assessment, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 4794-4802, 2021.
[25] Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett, editors. Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurIPS 2019, December 8- 14. 2019, Vancouver, BC, Canada, 2019. URL
neurips heurips.cc/paper
2019
[26] Jingyan Wang and Nihar Shah. Your 2 is my your 3 my 9: Handling arbitrary miscalibra- tions in ratings. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 864-872 872, 2019.
[27] Jingyan Wang, Ivan Stelmakh, Yuting Wei, and Nihar B Shah. Debiasing evaluations that are biased by evaluations. arXiv preprint arXiv:2012.00770 2020.
[28] Yining Wang, Liwei Wang, Yuanzhi Li, Di He, and Tie- Yan Liu. A theoretical analysis of ndeg type ranking measures. In Conference on Learning Theory, pages 25-54, PMLR, 2013.
[29] Hedayat Zarkoob, Hu Fu, and Kevin Leyton- Leyton-Brown. Report-sensitive spot-checking in peer- grading systems. In Proceedings of the 19th International Conference Autonomous Agents and MultiAgent Systems, pages 1593-1601, 2020.
12


Reviews:
[{"review_id": "ztRcxWUJNAg", "reviewer": "Reviewer_Emh1", "summary": "The paper addresses the problem of review calibration, i.e., to\nestimate true paper qualities from reviewer ratings, if they\nare monotonic functions in the true quantities. The authors model\nthe problem as an optimization problem that aims to minimize\ninput noise on the reviewer functions. The authors derive explicit\nlinear programs for the case of linear reviewer functions with\nand without noise and sketch further variants. They discuss differences\nof their approach to linear regression and matrix seriation.\nIn an experiment with synthetic data (and thus given ground truth),\nthey compare their method against two baselines from the literature\nand show improvements both in the case of linear reviewer functions\nbeing the groundtruth as also for more general reviewer functions.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very well written and a joy to read. The proposed method\nis plausible and the experimental results look very good, though only\nsynthetic data has been used.\n\nI have only three minor points with the paper:\n1. The proposed method is a pure optimization solution, thus the\n    paper maybe will be better positioned in an optimization conference?\n2. It might be practically interesting only for a small, special audience (PC chairs).  \n3. If there is no way to observe the groundtruth as the authors allude to\n   (\"if we already know the true qualities, peer reviews would not be needed\n   anymore\"), this would raise the question what exactly these \"true qualities\"\n   then are. Maybe the field of peer review calibration needs to find\n   some way to make them observable. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "bCLF31OpGU", "reviewer": "Reviewer_zLc7", "summary": "The paper has proposed a new flexible framework, least square calibration (LSC), for selecting top candidates from peer ratings. The major contributions are summarized as follows:\na.\tProposed a model which could handle different types of miscalibration in peer reviews, and also incorporate various levels of prior knowledge\nb.\tSelf-prove the effectiveness of LSC by both theoretically and empirically \n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\nIt is a good try to use LSC and other classes of linear scoring function for selecting top candidates for peer review, to the best of our knowledge, we do not see any previous papers have used this technique\n\nQuality\nThe paper has provided rigorous mathematical proof for the LSC and other stuff, however, there are not many experiments done by the author, for example one more ablation experiment on how prior knowledge is incorporated and how that will impact the results. And this is actually facing a chicken-and-egg problem, if you know the quality, peer review is not needed anymore, but if you do not know the quality, you will not know how the performances will be in the true data. It might be also good to make experiments on real data. It outperforms all previous systems by a lot, which is really impressive. The paper has been too long (9 pages) and exceeded the maximum allowed length (8 pages).\n\nClarity\nThe paper is well-organized, used well-selected notations, and has provided rigorous proof of the method, but sometimes it is hard for the readers to follow.\n\nSignificance\nThe paper has proposed LSC to address miscalibration problem for peer review which outperforms all previous models.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "XzphPw2vx4t", "reviewer": "Reviewer_PoXu", "summary": "This paper tackles the problem of calibrating ratings, focusing on the application of scientific peer review. The idea is to normalize reviewers who interpret the rating scale differently. If one reviewer tends to use low ratings and another tends to use generous high ratings, the calibration step will equilibrate them. The calibration requires the two reviewers to have some papers in common, or at least both have papers in common with a chain of other reviewers. The idea is that each paper has an inherent quality, each reviewer perceives that quality with independent zero-mean noise, and each reviewer translates that perceived quality into a rating using a monotone function. Given this model of how ratings are formed, the authors propose an optimization problem to find the hidden qualities, function parameters, and error rates that minimize the squared errors that reviewers perceive. They call this Least Square Calibration (line 133). The authors show how to do this for linear functions with and without noise, and prove some conditions under which the exact qualities can be recovered, including an if and only if connection to doubly connected review graphs with no noise. The authors also show how to do run the optimization for non-linear functions. The authors discuss similarities with linear regression and matrix seriation. The authors provide simulations to show that their method works.", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors address a real problem: how to calibrate ratings. The problem is hard because if one reviewer uses low ratings and another uses high ratings, it's hard to disentangle reviewer tendencies from paper qualities. If the papers the two reviewers rate are disjoint, it's especially hard, but not impossible, because the first and second reviewers may each have papers in common with a third reviewer or a chain of reviewers. The authors formulate the problem in a reasonable way. The solution is straightforward. I am not fully satisfied with the results for three reasons:\n\nFirst, the authors don't discuss enough the problem of overfitting. If the function hypothesis class is too large, the optimization will almost certainly overfit. For example, in the definition of LSC at line 133, if the function f is a complex non-linear function with many parameters, the optimization will be able to find a solution with all the epsilon error terms zero, all the x values arbitrary, and the function f memorizing a mapping from x to y. When the function is linear, overfitting is somewhat controlled. When the function is non-linear, the authors will need some sort of regularization.\n\nAFTER AUTHOR RESPONSE: the following second half of the previous sentence was not correct in my original review, as the authors' pointed out in their response: \", and the tests will need to be done in the usual ML way with the optimization being solved on different data than the tests are run.\"\n\nSecond, the empirical tests are simulations. For this application, testing on real data will be much more persuasive. Some peer review data is here: https://github.com/allenai/PeerRead/tree/master/data described in the paper https://arxiv.org/abs/1804.09635 . The data does include some numerical scores and includes accept/reject decisions that can be thought of as ground truth. There are other ratings data sets the authors can use. For example, data from peer grading systems should work, for example http://www.tml.cs.uni-tuebingen.de/team/luxburg/code_and_data/peer_grading_data_request_new.php . Here is another peer grading paper: https://www.cs.ubc.ca/~jrwright/mta/ . Data from review web sites might work too. Here is some review data: https://www.preflib.org/data/CD . There is no ground truth, so this may or may not work. One idea would be to use the average of all ratings as the ground truth and then try to recover the ground truth by calibrating a small subset of the users. A Mechanical Turk experiment would also be revealing and can be engineering to have a ground truth.\n\nAFTER AUTHOR RESPONSE: I am impressed with the new empirical results. Thank you for doing that with amazing speed. I am glad that the review process led to a better paper with both synthetic and real results. Still, this is only one test and it is very hard to evaluate a new test produced during the review process.\n\nThird, the empirical tests do not split the data into training set and test set, as far as I can tell. Perhaps I missed that. Without a proper split of training and testing, the authors cannot say whether their optimization overfit to the data.\n\nAFTER AUTHOR RESPONSE: Thank you for your response and clearing up my misunderstanding. I looked back at the paper again at the main optimization problem on line 133. This IS an unsupervised problem. So, my third complaint above about no train-test split in the experiments is not valid, as the authors point out. Sorry about that. \nHowever, I feel that my first argument is still valid. I don't agree that an unsupervised learning algorithm cannot overfit. For example, a clustering algorithm can get perfect results (by some metrics like purity or mutual information) by placing every data point in its own cluster: https://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-clustering-1.html This is a form of overfitting. Specifically, for this paper, in the definition of LSC at line 133, I don't see why some complex non-linear function f with many parameters couldn't be optimized to find a solution with all the epsilon error terms zero, all the x values arbitrary, and the function f memorizing a mapping from x to y. If the function class for f is too large (the analogy of setting k=n in k-means clustering), the optimization can memorize an answer that has all error terms zero.\n\n\nI had one more question:\n\nIs Definition 2.1 (Linear Scoring Function) only for the linear function without noise? What happened to the independent zero-mean noise? Is it zero? What is the linear scoring function with noise? Should it be a(x+epsilon)+b?\n\nHere are some minor writing suggestions/corrections:\n\nthere exists cardinal scores estimators perform better than any ordinal ranking estimators\nthere exist cardinal score estimators that perform better than any ordinal ranking estimators\n\n320: the same number of reviews\n320: the same number of reviewers\n\nassigned scoring function from pre-specified family\nassigned a scoring function from a pre-specified family\n\nbayesian model\nBayesian model\n\nReference [24] has no publication venue", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "N5YO22_8Gxb", "reviewer": "Reviewer_o3ak", "summary": "This paper studies calibration in the peer review process. Unlike most previous work, in this paper, the structure of reviewer miscalibration needs to only be monotone (in the true score of the paper being reviewed).  The authors also study the cases where: miscalibration is linear, and there is added noise.  The main contribution is framing the calibration problem as a (specific) functional optimization problem. Theoretically, they prove that optimal solutions to their optimization problems yield the correct underlying ordering of the papers (by true score). In comparison to 3 baseline algorithms the proposed approach tends to perform best in simulated experiments.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Originality*:\nFraming the calibration problem as functional optimization seems novel to me and is interesting.  In experiments it also tends to yield superior results (i.e., precision/GAP) than other competing methods. Similarly, while subtle, I think the idea of putting the noise term _inside_ the score generation (equation 1), is quite useful and makes the problem easier to solve (than putting the noise term outside). I don’t see a problem with this approach immediately; assuming it is not problematic, this could be used in further research on calibration. \n\nAdditionally, I found remark 3.5 especially interesting. I was unaware that additional reviews are requested in order to achieve better calibration (I thought additional reviews were requested for other reasons).  Authors: can you give a citation or anecdotal evidence of this? Either way, the remark implies a strategy of holding out some reviewers and assigning them after an initial set of reviews for the sake of better calibrating the reviewer scores.\n\n\n*Quality*: \nThe work seems technically sound: the “if” direction of Theorem 3.4 seems correct; the “only if” direction I’m a bit confused about. Theorem 3.6 also seems correct.  All functional optimization problems outlined seem correct as well.\n\nIn the experimental setup: M=1000 and N=1000 seems unrealistic. Why not use the exact parameters from ICLR? I’m curious about whether/how these parameters affect the results.  Moreover, I’m interested in how the distribution of paper (true) scores affect the results (in terms of precision/gap). I’m also curious about other evaluation metrics: how many papers would be wrongly accepted/rejected when using each of the baselines? The precision/gap get at this, but it would be illuminating to calculate statistics like this for each method. \n\nFinally, in the experimental section, it would be useful to describe why the proposed method outperforms the other baselines in each case, especially in the case of linear functions (with/without noise).\n\n*Clarity*: \nOverall, the paper was clear. However, some details related to clarity could be improved. For example: what is the range of true scores, $x*$? Of the scores x in the LP?  In the LSC function with linear scores and no noise: it was at first confusing to me that the scoring functions do not appear; consider adding them or explaining this in the text. In definition 3.1: I’m a bit confused if $x(i)$ is a linear function of the true score and $y(i)$ is a linear function of $x(i)$.  Also, can you explain why the first constraint in CP 4 is necessary? Why does Theorem 3.6 only imply that the true item ordering is recoverable but not the scores?\n\nAt first glance, Section 4 seems a bit superfluous.  That being said, I found 4.1 helpful. Perhaps some of this information could be condensed and added into previous sections. This would give room for more experimental results.\n\nIn the experiments section, k is referenced before it is defined.\n\nIn Figure 1: why does the precision of Average go down with increasing fraction of linear functions?  Moreover, it seems like EQ performs much worse when noise is added; can you explain why?\n\nThere are grammatical errors in the proof sketch of Theorem 3.4 (there are extra “the”s).\n\nAlgorithm box 1 may not be necessary (just describe it in words and put the algorithm in the appendix). \n\n\n*Significance*:\nI found the result from 3.4 interesting, however, I’m unsure of how technically novel it is. Moreover, I’m curious as to how often we’re likely to get a doubly-connected component that covers all papers in practice.  I think it would be worth it to check this using real data and standard (max-similarity) matching. \n\nApplying function optimization to reviewer calibration is reasonable and I’d be very curious to see how this works in practice.  Similarly, I think it would be worthwhile for program chairs to experiment with this style of calibration in practice.\n\n\nEDIT AFTER AUTHOR RESPONSE:  Thank you authors for clearing up some of my confusion.  Implementing the changes you identified in your response (e.g., writing changes, more realistic simulation and extrapolation of ethical concerns) will improve the paper. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There are a handful of limitations with this work.  First and foremost, the monotonicity requirements on the score generation. While this is a significant limitation, a) this could be reasonable in practice and b) virtually all related work makes the same assumption.\n\nAnother limitation is the assumptions required on the type of scoring functions (linear/monotonic).  This seems to be difficult to judge a-priori and important: e.g., the QP approach outperforms the proposed LSC-monotone approach in the noiseless case.  That being said, the noiseless case is perhaps a bad approximation of reality. \n\nAs mentioned above, the expectation of doubly connectedness in the review graph is a limitation. Do all the results require this? Do all the simulations guarantee that the review graphs are doubly connected? And, if so, what happens if the review graphs break this assumption?\n\nFinally, I’d like to better understand the design choice of applying noise to the papers rather than the reviewers. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {"ethical_concerns": "This paper does not include a statement related to ethics but I believe it is worth thinking about/discussing. Applying this framework could be problematic if discovering incorrect calibration functions, and leading to the rejection of papers that would otherwise be accepted. It might be worth it to analyze the number of flipped accept/reject decisions when compared to average vs. true score--or some similar analysis. In a similar spirit, it’s worth it to think through and describe when a naive application of the proposed approach would fail and what the consequences would be."}}]

Author rebuttal:

Meta-review: This paper addresses miscalibration in peer review. Miscalibration can cause problems with the fairness of the peer review process, but so far not much has been achieved in terms of theoretical algorithms that also have practical appeal. Some previous papers study simple linear models and another study considers arbitrary or adversarial miscalibration. This paper fills in this void by considering general classes of miscalibration models. It proposes an interesting "LSC" method, provides theoretical results to back it up, and simulations to investigate it further. The rebuttal, in response to some reviewers' comments, also provides experiments on real-world data and their algorithms work well here. This is a good contribution to the field and I recommend acceptance.

Important: In their camera ready version, the authors should incorporate the various points discussed with the reviewers or with me during the review process. We hope that this review process has helped the authors improve the paper.

Summary of reviews:
- Reviewer o3ak provides a detailed review, regards the paper as novel and interesting
- Reviewer zLc7 makes some initial criticisms in their initial review. The rebuttal addresses the objective parts of these criticisms but the reviewer did not show up after the rebuttal (I had sent multiple pings).
- Reviewer PoXu provides a detailed review. They make several criticisms in their initial review, and the objective parts of these criticisms are addressed in a 3-way discussion between me, the authors, and the reviewer.
- Reviewer Emh1's review has only minor points of concern

Decision: Accept (Poster)