Least Square Calibration for Peer Reviews

1 Introduction . Peer review has been essential to the advancement of academic research . In a typical conference peer review process , the submitted papers are assigned to a set of external reviewers and will receive cardinal scores from them . The final decision of which papers get accepted to the conference is usually determined by the average score received for each paper . However , during such a process , different reviewers may have different standards and biases in their evaluations . For example , the same paper may receive a high score ( e.g. , 8 out of 10 ) from a lenient reviewer and a low score ( e.g. , 2 out of 10 ) from some strict reviewer . As a result , the average score may not always reflect the true quality of a paper . This issue is known as miscalibration , and is a prevalent problem in many other online review systems as well such as TripAdvisor or Yelp . In order to ensure a fair and unbiased outcome , it is crucial to the concept and to the success of a peer reviewing system to have an efficient and effective calibration algorithm . There has been a body of works attempting to address the miscalibration issue in peer reviews . Most of these works can be categorized into two directions . The first line of works relies on making simplifying assumptions on the form of miscalibration . For example , [ 19 , 5 , 10 , 16 , 27 ] ⇤Equal Contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . model the miscalibration as one-parameter additive bias for each reviewer ; [ 19 , 21 ] assume each reviewer ’ s scoring function to be a linear function . Many of these works are able to accurately learn the unknown parameters of the miscalibration due to its simple structure . However , these strong structural assumptions on the miscalibration are often too strong for these results to be widely applicable . The second line of research [ 20 , 9 , 13 , 17 , 18 , 3 ] allows the miscalibration to be arbitrary , and only relies on ordinal ranking information obtained from cardinal scores to calibrate . A recent work [ 26 ] bridges these two lines of research by showing that even when the reviewers ’ functions are arbitrary , there exist cardinal score estimators that perform better than any ordinal ranking estimators . However , their work is still pertained to worst case assumptions , and is restricted to the scenario where each reviewer only reviews a pair of items . It remains an important open problem to design calibration models and algorithms that can work with weaker simplifying assumptions ( e.g non-linear functions ) in a practical scenario . Summary of Contributions . In this work , we propose a new optimization-driven unsupervised learning framework , least square calibration ( LSC ) . As the name suggests , the framework leverages the natural and powerful least square method for handling different types of miscalibrations in peer reviews . A key strength of our framework is its flexibility in incorporating various levels of prior knowledge , which consequently results in different qualities of calibration . Specifically , we capture the system ’ s prior knowledge about reviewers ’ scoring as a hypothesis class of scoring functions . Depending on how much the system knows about reviewers , the hypothesis class can be as “ rough ” as all monotone functions , or as specific as linear functions , convex/concave functions , Lipschitz continuous functions , or even their mixtures . Our framework can be tailored to accommodate all these hypothesis classes and moreover , cope with the reviewers ’ noisy perception of true item qualities . We demonstrate the effectiveness of the LSC framework both theoretically and empirically . We start our theoretical analysis from the hypothesis class of linear scoring functions . When reviewers ’ scores are generated by arbitrary linear functions of true item qualities without noise , our main result is a necessary and sufficient characterization about when LSC can ( essentially ) perfectly recover true item qualities . Interestingly , it turns out that perfect recovery is fully captured by a novel graph connectivity notion for a carefully constructed reviewer graph , which we coin double connectivity . This result reveals useful insights about what kind of reviewer-item assignments may be good for calibration , which is further justified in our experiments . When reviewers perceive item qualities with noises , we show how to reduce the LSC framework to a tractable convex optimization program when scoring functions are linear . We then extend this technique to accommodate other function classes including monotone , convex , or Lipschitz continuous functions . Our theoretical analysis is concluded with a discussion about the connection of LSC to other well-known problems such as regression and matrix seriation . Empirically , we test the performance of LSC via extensive experiments . We show that LSC consistently outperforms previous methods in various problem setups . The experiments also demonstrate that the calibration quality of LSC improves as our prior knowledge improves and LSC is robust to reasonable amount of misspecified prior knowledge . We hope these encouraging findings can motivate more future studies about how to identify more accurate class of reviewers ’ scoring functions . 2 Problem Formulation . In this section , we will first formally describe the peer review problem and then introduce our calibration approach . We use conference peer review as an illustrative example , though our model and methods can be trivially generalized to many other peer review systems . The Peer Review Problem . In a peer review process such as an academic conference , there are N items and M reviewers . By convention , we use [ N ] = { 1 , · · · , N } to denote the set of items and [ M ] = { 1 , · · · , M } to denote the set of reviewers . We assumes that each item has an inherent ground truth quality x⇤ ( i ) . Throughout the paper , the superscript notation “ ⇤ ” will always be used to denote true qualities . For simplicity , we assume that there are no ties and the item qualities are all distinct from each other . Each reviewer j reviews a subset of the items Ij ✓ [ N ] . For now , we allow Ij to be an arbitrary subset , which can even be a singleton or the entire set of items [ N ] . We will specify later under what conditions about { Ij } j2 [ M ] our algorithms will work . We denote I ` j 2 [ N ] as the index of the ` ’ th item reviewed by reviewer j . Scoring Functions and Score Generations . Each reviewer j 2 [ M ] is characterized by a monotone scoring function fj : R ! R. Like most previous works in this space [ 10 , 21 , 5 , 16 , 27 ] , we assume that reviewers ’ scores are generated by applying her scoring function to a noisy perception of the item ’ s true quality . Formally , reviewer j ’ s score about item I ` j , denoted by y ` j , is generated as follows Score Generation : y ` j : = fj ( x ⇤ ( I ` j ) + ✏ ` j ) ( 1 ) where fj is reviewer j ’ s scoring function and ✏ ` j is an independent zero-mean noise . For convenience , we always assume the scores for each reviewer j is ordered increasingly such that y1j < y2j · · · < y |Ij | j . Design Goal . We are interested in designing an item selection algorithm . The algorithm takes as input the item assignment { Ij } j2 [ N ] for a set of items [ M ] and a set of reviewers [ N ] , the reviewers ’ scores for each of their assigned item { y ` j } j2 [ M ] , ` 2 [ Ij ] , and an additional threshold parameter n N . The goal of the algorithm is to output a set S of n items that best matches the top n items according to the ( unknown ) true qualities x⇤ . In the experimental section , we will describe multiple metrics to evaluate the performance of our algorithm . Restricted Classes of Scoring Functions . Note that when defining reviewers ’ scoring functions , we do not make any assumptions on the properties of these functions other than monotonicity . This is to ensure that our framework is flexible and can handle a wide range of problems . However , in many situations , the designer may have some additional prior knowledge and infer more structures about the scoring functions . The reason of assuming some restricted class of scoring functions is multifold . First , such functions may be a good proxy or approximation of the true scoring functions based on our prior knowledge . Second , it may help us to avoid the issue of overfitting2 since allowing arbitrary scoring functions may be able to fit the review scores well but leads to poor generalization and make the result prone to perception noise . For example , many previous works [ 21 , 10 , 5 , 16 ] have focused on the basic model of linear scoring functions , defined as follows . Definition 2.1 ( Linear Scoring Function ) . A linear scoring function of reviewer j has two parameters : aj 0 and bj 2 R and is of the form fj ( ex ( = x+ ✏ ) ) = ajex+ bj . In addition to linear scoring functions , one may also consider broader class of scoring functions when less prior knowledge is available or when we want to make less stringent assumptions . For example , our framework can also handle scoring functions that are convex or concave or a mixture of these functions.3 However , we emphasize that it is not our purpose to justify which of these classes of scoring functions are more realistic since this answer would depend on the concrete domains of the problem . Instead , our goal is to develop a generic learning framework that can integrate any of such prior knowledge to improve the calibration . That being said , we believe it is an important future research direction to design heuristics , regularizations and other methods to choose the hypothesis class for better calibration results as well as to understand the potential issue of overfitting for hypothesis class that are too broad . 3 The Optimization-Driven Framework . Central to our approach is a framework to recover the underlying item quality x that can properly “ explain ” the observed review scores given the assumptions about the scoring function structures . Note that our recovered qualities may be up to some constant shifting or re-scaling from the true qualities . However , as long as the shifting and re-scaling preserve monotonicity , this will not affect the accuracy of our selection of the best n items . Following the notation in machine learning , let H denote a hypothesis class of scoring functions . For example , for linear scoring functions , the hypothesis class is H = { f : f ( x ) = ax+ b for some a 0 , b 2 R } . We denote this linear scoring function class as HL for convenience . More generally , H may be any class of functions depending on designer ’ s prior knowledge of these scoring functions . Given a hypothesis class H , our goal is to find scoring functions f1 , · · · , fM 2 H for all reviewers , such that the distance between the true qualities x and qualities recovered by these selected scoring 2Overfitting here is in the similar sense to degenerated cases such as setting k = n in k-means clustering of n samples , or in low-rank matrix completion where k is the target rank , n is the matrix size . 3A scoring function f is convex if for any x1 , x2 2 R and for any t 2 ( 0 , 1 ) , we have f ( tx1 + ( 1 t ) x2 ) tf ( x1 ) + ( 1 t ) f ( x2 ) . Function f is concave if f is convex . functions is minimized . In this paper , we use the Euclidean distance . This leads to the following general optimization-based framework , which we term Least Square Calibration ( LSC ) . Note that in general LSC is a functional optimization problem ( FOP ) with : ( 1 ) functional variables f = { fj 2 H } j2 [ M ] ; ( 2 ) item quality variables x = { x ( i ) } i2 [ N ] ; ( 3 ) noise variables ✏ = { ✏ ` j } j2 [ M ] , ` 2 [ |Ij | ] . Least Square Calibration ( LSC ) min x , f , ✏ MX j=1 |Ij |X ` =1 ( ✏ ` j ) 2 ( 2 ) s.t . y ` j = fj x ( I ` j ) + ✏ ` j 8j 2 [ M ] , ` |Ij | where the set cardinality |Ij | is the number of items reviewer j reviews . LSC tries to find the scoring functions and true qualities that best fit all the review scores in terms of the least square error . The rationale behind LSC can be seen from several perspectives . First , fix any ( unknown ) parameters x and f , define random variable Y ` j = fj x ( I ` j ) + ✏ ` j with randomness inherited from the noise ✏ ` j . Consider the following parameter estimation problem4 — given observed realizations of Y ` j as y ` j , find parameters x and f to maximize the likelihood of the observation . It is not difficult to see that the optimal solution to LSC is the maximum likelihood estimator , when ✏ ` j is i.i.d . Gaussian noise with mean 0 . Second , it turns out that LSC can also be viewed as a variant of the classic matrix seriation problem . In Section 4 , we will discuss this connection in more details as well as the connection of LSC to linear regression .

Summary: This paper tackles the problem of calibrating ratings, focusing on the application of scientific peer review. The idea is to normalize reviewers who interpret the rating scale differently. If one reviewer tends to use low ratings and another tends to use generous high ratings, the calibration step will equilibrate them. The calibration requires the two reviewers to have some papers in common, or at least both have papers in common with a chain of other reviewers. The idea is that each paper has an inherent quality, each reviewer perceives that quality with independent zero-mean noise, and each reviewer translates that perceived quality into a rating using a monotone function. Given this model of how ratings are formed, the authors propose an optimization problem to find the hidden qualities, function parameters, and error rates that minimize the squared errors that reviewers perceive. They call this Least Square Calibration (line 133). The authors show how to do this for linear functions with and without noise, and prove some conditions under which the exact qualities can be recovered, including an if and only if connection to doubly connected review graphs with no noise. The authors also show how to do run the optimization for non-linear functions. The authors discuss similarities with linear regression and matrix seriation. The authors provide simulations to show that their method works.