Integrating Rankings into Quantized Scores in Peer Review
Authors: Yusha Liu
Yichong Xu
Nihar B Shah
Aarti Singh
Venue: Accepted by TMLR ()

Abstract: In peer review, reviewers are usually asked to provide scores for the papers. The scores are then used by Area Chairs or Program Chairs in various ways in the decision-making process. The scores are usually elicited in a quantized form to accommodate the limited cognitive ability of humans to describe their opinions in numerical values. It has been found that the quantized scores suffer from a large number of ties, thereby leading to a significant loss of information. To mitigate this issue, conferences have started to ask reviewers to additionally provide a ranking of the papers they have reviewed. There are however two key challenges. First, there is no standard procedure for using this ranking information and Area Chairs may use it in different ways (including simply ignoring them), thereby leading to arbitrariness in the peer-review process. Second, there are no suitable interfaces for judicious use of this data nor methods to incorporate it in existing workflows, thereby leading to inefficiencies.
We take a principled approach to integrate the ranking information into the scores. The output of our method is an updated score pertaining to each review that also incorporates the rankings. Our approach addresses the two aforementioned challenges by: (i) ensuring that rankings are incorporated into the updated scores in the same manner for all papers, thereby mitigating arbitrariness, and (ii) allowing to seamlessly use existing interfaces and workflows designed for scores. We empirically evaluate our method on synthetic datasets as well as on peer reviews from the ICLR 2017 conference, and find that it reduces the error by approximately 30% as compared to the best performing baseline on the ICLR 2017 data.

Published in Transactions on Machine Learning Research (09/2022)
Integrating Rankings into Quantized Scores in Peer Review
Yusha Liu
yushal@es.cmme eduu
Machine Learning Department Carnegie Mellon University
Yichong Xu
gichong.zu@mmcrossofff
Microsoft Cognitive Services Research
Nihar B. Shah
nihors@cs.cmm cmau. edu
Machine Learning Department, Computer Science Department
Carnegie Mellon University
Aarti Singh
aarti@cs. marti@cs.cmu.co cmnu. edu
Machine Learaing Department
Carnegic Mellon University
Reviewed on OpenReview: https: // openrevie.. nor/forum? id- Kb 1bbuuLLa
Abstract
In peer review, reviewers are usually asked to provide scores for the papers. The scores are then used by Area Chairs or Program Chairs in various ways the decision-making process. The scores usually elicited in quantized form to accommodate the limited cognitive ability of humans to describe their opinions in numerical values. It has been found that the quantized scores suffer from a large number of ties, thereby leading to a significant loss of information. To mitigate this issue, conferences have started to ask reviewers to additionally provide a ranking the papers they have reviewed. There are however two key challenges. First, there is no standard procedure for using this ranking information and Area Chairs may use it in different ways (including simply ignoring them), thereby leading to arbitrariness in the peer-review procoss. Second, there data nor methods to incorporate it in existing workflows, thereby leading to inefficiencies.
suitable interfaces for judicious use of this
We take a principled approach to integrate the ranking information into the scores. The output of our method is an updated score pertaining to each review that also incorporates the rankings. Our approach addresses the two aforementioned challenges by: (i) ensuring that rankings are incorporated into the updated scores in the same manner for papers, thereby mitigating arbitrariness, and (ii) allowing to seamlessly use existing interfaces and workflows designed for scores. We empirically evaluate method synthetic datasets as well peer reviews from the ICLR 2017 conference, and find that it reduces the error by approxinately 30% as compared to the best performing baseline on the ICLR 2017 data.
1 Introduction
Many applications involve people evaluating large number of items in distributed fashion. An important and prominent such application, which is the focus of this paper, is peer review of papers in scientific conferences. A typical way of collecting reviews is through quantized scores, that is, where reviewers are asked to provide scores from constant number of quantization levels. For example, reviewers for conferences are often asked to provide their opinions on papers in the format of five-level or ten-level Likert items, which are used to evaluate the qualities of submitted papers.
A drawback of such quantized scores that there exist a large number of ties such quantized scores, due to the constant number of quantization levels well as the respondents' tendency to give equal values in

---

Published in Transactions on Machine Learning Research (09/2022)
the absence of prompt for relative relationships (Feather, 1973). For example, recent study (Shah et al., 2018) of peer-review data from the NeurIPS 2016 conference found that among all instances where a reviewer reviewed a pair of papers, the pair of review scores provided by the reviewer were tied in more than 30% of such instances. Such a large proportion of ties were found to cxist in the scores for all four criteria elicited from reviewers, and the number of ties was even higher when restricting attention to only the top and middle-quality papers. Such ties result in a loss of information, thereby contributing to the difficulty in making decisions regarding the acceptance of papers.
Despite the apparent drawbacks, there are strong reasons that quantized scores widely used format instead of continuous- scores to clicit the opinions of reviewers. This is because of the limited ability of humans to describe stimuli with numerical values (Miller, 1956). Psychometric studies have discussed the appropriate number of response alternatives when eliciting responses from humans and suggested that it remain a small constant (Lietz, 2010; Jones & Loe, 2013).
An alternative form of evaluation comprises rankings. Ranking information alone has been demonstrated to be robust way to collect information (Rankin & Grube, 1980; Douceur, 2009; Shah et al., 2013; 2016b). For example, crowdsourcing experiments (Shah et al., 2016b) demonstrate the reliability of answers in form of pairwise comparisons, which incur lower per-sample error as compared to cliciting numerical values.
Scientific conferences, which face a growing number of submissions each year, are increasingly trying to collect additional information by asking reviewers to report rankings in addition to scores (Soergel et al., 2013). Among recent computer science conferences, the NeurIPS 2016 conference asked each reviewer also rank the papers they were reviewing (Shah et al., 2018). This was an experiment that provided a sanity check about rankings in peer review and also identified several benefits of collecting rankings. The ICML 2021 conference, as well as the recent AAAI 2023 conference collected rankings in addition to scores from reviewers on their assigned papers. Note that during the review process, a majority of the reviewers' time is spent on reading and evaluating the papers. As a result, generating rankings in addition to the traditional scores may take only little additional time.
In Figure la we illustrate the standard interface used by Area Chairs in peer review, augmenneed with the ranking information (in the "rankings" column). An important aspect of the standard interface, extensively used by chairs in their workflow, is the ability to sort papers via the minimum, maximum, average received scores or via the spread of the scores. The augmented ranking information shows the (partial or total) rankings provided by reviewers, where papers not being handled by that Area Chair are replaced with a * symbol (as done by ICML 2021). There are multiple challenges of such an interface. First, such ranking information is incompatible with commonly used workflow elements such as sorting according to scores, so any such operations will omit ranking information. Second, it not clear how to efficiently extract information from the rankings under such an cluttered interface. As a consequence, the varying use or lack of use of ranking information by different Area Chairs can add to the arbitrariness in peer review. Thus, while the elicitation of ranking evaluations introduces another kind of information for chairs to use, the question of how to judiciously use this data has remained open.
Our goal is to allow the chairs to use the ranking information in addition to the scores while not disrupting their workflow. To this end, we design a method to integrate the reviewer-provided rankings into the scores. One may think of this approach as dequantizing the scores. In our problem formulation (Section 3), the output is real-valued score for each review, where these real-valued scores combine the reviewer-provided (quantized) scores and rankings. We refer to these real-valued outputs as the "dequantized scores" and illustrate the interface with these dequantized scores in Figure 1b. As shown in Figure 1b, these dequantized scores, unlike the rankings, can now be incorporated in the standard workflow for Area Chairs or Program Chairs, allowing them to seamlessly perform the tasks that they conventionally perform on the original quantized scores (such as sorting by the average or the spread of the scores for each paper).
Our contributions. The main contributions of this paper are as follows.
(1) We identify and formulate the problem of combining rankings and quantized scores, focusing specifically on dequantization of the scores for every review, rather than aggregating the quantized scores and/or rankings given by reviewers to estimate score for each paper. We provide detailed motivations for dequantizing the
2

---

Published in Transactions on Machine Learning Research (09/2022)
Overall Evaluation (Scores)
Paper
Tible
Reviowers
Scores
Rankings
Min
Max
Avg
Spread
1iter...
G
e.g.
The Hobbit
[RT]]"> 134 (R4) > 184,
134
0.47
609
The Fellowship of the Ring
Rt: RS; F8
8: 5:55
[16) >> 639
0
1200
The Two Towers
R2; RG; R7
6, 6:
[R2]: 1209 >
0
King
> 104, [A5]: 6390
6390
Pii: RS; R8
7: 8:
6.67
0.47
[R8]
(a)
Iuterface with quantized scores and rankings.
Overall Evaluation (Dequantzed Scores)
Paper ID
Title
Reviewers
Original Scores
Dequantized Scores
Min
Spread
eg
fiter...
fitter..
fiter..
fiter...
e.gi
<3
egg.
e.g. <3
egg <3
5.77; 6.54;
5.77
890
The Followship of the Ring
R1; RG;
5.
4.86; 5.17
4.88
5.17
4.98
0.2
1209
The Two Towers
R2: RS;
8:0:
6.27; 6.18: 6.48
6.18
6.48
8.31
0.18
6290
Return the King
7;6;7 7.10;
6.02
8.71
0.49
(b) Interface with dequantized scores.
Figure 1: An illustration of the envisaged interfaces for the Area Chairs, a conference management system (Microsoft CMT) used commonly by computer science conferences for peer-review. The top figure shows the interface with separate quantized scores and rankings reported by reviewers. The rankings the top figure are processed in the same manner as in the ICML 2021 conference, precisely, papers that are not handled Area Chair replaced with symbols for confidentiality. The bottom figure shows the interface with proposed output format of dequantized scores. Each represents paper that the Area Chair is handling, while the columns correspond to relevant information of the paper.
scores in Section 4.1. Our work is applicable even if each reviewer reviews only a small subset of the papers, and also if reviewers may only return a partial ranking of the assigned papers. For concreteness, we focus on the application of peer-review, while noting that the problem and solution may also apply other settings where both rankings and quantized scores are available.
(2) We propose comppitatonnally-fffccet algorithm that outputs real-valued dequantized score for every review in the assignment.
Our approach based on set design principles we outline subsequently. We make no parametric assumptions on the data generation process, nor the existence of ground truth quality scores and global rankings.
A part of our approach is inspired by isotonic regression, and we also provide connections to esti- mation under the Thurstone model and the balanncotrank-esiimtiinn algorithm (Wauthier et al., 2013, Section 4.1).
Our algorithm includes purely data-driven "Quantization Validation (QV)" method for hyperpa- rameter selection due to the absence of the ground truth.
(3) We evaluate the empirical performance of our proposed algorithm on both synthetic data and a semi- synthetic dataset based on real review scores from the ICLR 2017 conference. As motivated subsequently in Section 4.2, the (Kendall-tau) ranking error the metric of interest. We compare the proposed
3

---

Published in Transactions on Machine Learning Research (09/2022)
algorithm to two baselines: quantized scores and BRE-adhusted-scores which are formally introduced in Section 5.2. We find that our algorithm incurs about 28% lower error as compared to the best performing baseline in the ICLR 2017 data.
2 Related Work
There is a long line of literature (Wauthier et al., 2013; Eriksson, 2013; Braverman et al., 2016; Shah et al., 2016;; Shah & Wainwright, 2017; Shah et al., 2016b; Mao et al., 2017; Negahban et al., 2017; Pananjady et al., 2017; Agarwal et al., 2018; Makhijani & Ugander, 2019; Wang et al., 2020) in the domain of ranking from pairwise comparisons with various modeling assumptions. Methods from this domain take compar- ison outcomes between pairs of items input, and output estimates of an underlying global ranking comparison probabilities matrix. The problems investigated in this domain, however, are fundamentally different from ours with major distinctions in both the input and output of the problem. In our setting, the algorithm needs to effectively incorporate scores in addition to rankings, while the above methods cannot be trivially adapted to take score values as input. In settings such as peer review, scores cannot be neglected, as the ranking information very limited and sparse. Additionally, the number of comparisons per item needed in most prior work in ranking from pairwise comparisons grows with the number of items (sometimes logarithmically but often linearly). In practice, however, there are often only 3 to 6 reviews per paper in most peer-reviewed conferences, and the number of comparisons per paper is small constant. Mao et al. (2017) allow for the number of comparisons to be constant, but still use at least tens of comparisons per item in their simulations, which remains impractical for the peer-review setting. Furthermore, unlike their goals, we make a deliberate design choice to not aim to output global ranking or comparisons probabilities of the papers, but instoad focus on estimation of doquantized scores for reviewer-paper pairs. This design choice motivated in Section 4.1.
Some recent works (Hopkins et al., 2019; Zeng & Shen, 2020; Hopkins et al., 2020; Xu et al., 2020a; 2017; 2020b;e; Somers et al., 2017) develop approaches to use ranking information in addition to labels (scores) in supervised tasks such as classification, regression, and optimization. However, there are crucial differences between their settings and ours, such as the preservation of distinct reviewer evaluations, instead of pool- ing the data together, which prohibit any direct translations of those works to our setting. These works consider the generalization setting by building general predictive models from training data that then apply to incoming test data. Whereas our work is in the transduction setting, where we derive the output scores directly from the input quantized scores and rankings.
The kind of data considered in the paper by Ailon (2010) is closer to our setting despite their goal of deriving a global ranking, which considers two kinds of input: a total ranking for only a few top-ranked items and quantized scores. However, the method is comparison-based and therefore treats the quantized scores as (partial) rankings. This step thus leads to a significant loss of information. For instance, given a total of two papers X and Y, a reviewer giving a score 8 to paper X and soore to paper Y very different from giving 4 to X and 3 to Y, but the model would only retain the relative ranking between X and Y and not distinguish these two cases. In our setting, such data is equivalent to having only partial rankings but no scores.
The paper most closely related to our setting is a concurrent and independent work by Pearce & Erosheva (2022), which also considers a transduction setting with both ranking and score data. They propose a model termed the Hallows-Binomin model, parametric model that jointly captures the scores and rankings. They estimate the model parameters via maximum likelihood cstimation and propose two computationally--fffiicn algorithms based on A* tree search. They present theoretical results including properties of the Maximum likelihood estimator (MLE) for model parameters. On the empirical front, they fit the model on real-world grant panel review datasot where 6 judges each scored all of the 18 proposals and ranked their top 6. They examine the results manually in absence of ground truth and show that the estimated model parameters successfully capture information from both scores and rankings.
The work of Pearce & Erosheva (2022), however, differs from ours in several critical ways. First, their method and analysis require that each reviewer reviews the entire set of objects to be evaluated. While this condition may be met in small grant proposal panels such as the American Institute of Biological Sciences
4

---

Published in Transactions on Machine Learning Research (09/2022)
grant proposal review studied in Pearce & Erosheva (2022), this is impractical in conference peer-review that has thousands of submitted papers, which is the primary focus of our work. Second, they assume that each reviewer provides a top-K ranking. We allow reviewers to provide comparisons between any arbitrary subsets of assigned papers (Section 3), which eventually constitutes a partial ranking different from that in Pearce & Erosheva (2022). Third, the Mallows-Binomial model assumes the existence of true underlying quality score for each paper and consequently of a global ranking of the papers, whereas our approach deliberately aims to avoid assumptions about the ground truth scores and about ranking over the papers. Fourth, their model assumes that rankings and quantized scores are independent conditioned on the model parameters. On the contrary, we assume that the rankings provided by any reviewer are consistent - and hence strongly dependent with the quantized scores provided by that reviewer. Our setting occurs more naturally in peer-review, where reviewers report the two lypes of information at the same time as their final evaluations. Fifth, they pool the reviewers' data together to obtain a final quality score for each paper. On the other hand, our approach socks to provide flexibility to the Area Chairs and Program Chairs on aggregating the reviews for any paper, and hence we output score for each review.
Finally, there has been flurry of works that address various other problems in peer review, such bias (Tomkins et al., 2017: Stelmakh et al., 2019a; Manzoor & Shah, 2021), miscalibration (Flach et al., 2010: Roos et al., 2011; Wang & Shah, 2019; Tan et al., 2021), subjectivity (Lee, 2015; Noothigattu et al., 2021), dishonest behavior (Vijaykumar, 2020; Littman, 2021; Wu et al., 2021; Jeemen et al., 2020; Xu et al., 2019; Dhull et al., 2022), and others. See Shah (2022) for a survey, In particular, Noothigattu et al. (2021) address the problem of reviewer subjectivity, where their proposed output is similar to that we consider here an updated score pertaining to each review.
3 Problem Setting
Consider a set of P papers indexed by p E (P), and a set of R reviewers indexed by r E [R]. Tet. A denote the set of assigned reviewer-paper pairs: A = {(r,p) E [R] x [P] reviewer r reviews paper p}. For every paper p assigned to reviewer r, the reviewer reports quantized score 3rp E Z bounded by a pre-specified interval. In this work, we assume that higher score value indicates a more favorable review. In our experiments, both the number of papers assigned to a reviewer and the number of reviews received by paper are constants, which is the case in practice in peer-review, as opposed to scaling with P or R.
In addition, the reviewer reports a partial ranking ny, which a partial ordering over the set of papers that is assigned to this reviewer. For a pair of papers (p, P') reviewed by reviewer r, we use p >r p' to mean that reviewer r ranks paper p above paper p' in the partial ranking Ty provided by reviewer r. The partial ranking n, can be then represented the set म, (p.p') (r.p) p) E A, (r, E A.p > p'}. ). We assume that for every reviewer, the reported rankings are consistent with the scores, that is, the reviewer may give the ranking > p' only when the reported scores satisfy Erp 2 Arp'
Observe that under our problem formulation, reviewers can choose to report a total ranking of assigned papers, but also have the freedom to not compare some pairs of papers. In peer review, it not always reasonable to ask for a binary comparison between every pair of papers. For example, as pointed out in Rogers & Augenstein (2020), it can be difficult to make a comparison in situation with "incomplete evaluation in one borderline paper vs narrow applicability of another". Allowing partial orderings prevents reviewers from being forced to make such "apples-to-oranger comparisons (Rogers & Augenstein, 2020).
We now describe our objective given the inputs of quantized scores and rankings. As introduced in Section 1 and detailed in Section 4.1, given the quantized scores and rankings from the reviewers, our goal is to merge the two sources of information into a single source, in a manner that can seamlessly assist the humaa experts. Our designed algorithm thus aggregates the pwieewr--rovider sources of information into dequantized and continuous scores. For every (r,p) E A, we denote the output dequantized score as Trp E RR
For any positive integer m. we use the standard notation [m] {1, 2,... m....
5

---

Published in Transactions on Machine Learning Research (09/2022)
4 Main Results
We first describe our three design principles in Section 4.1. Then in Section 4.2, we present our proposed algorithm designed based on these principles. In Section 4.3, we draw interesting connections between our algorithm and simpler algorithms under some extreme cases, as well as a connection to maximum likelihood estimation under the Thurstone model. Finally, we present our data-driven Caarreennnuuttiinn 'Quantization Validation' method for hyperparameter selection in Section 4.4.
4.1 Approach and Design Principles
In this section, we introduce three principles we follow in designing our algorithm.
The first principle pertains to our approach towards the goal of the algorithm. Unlike many prior works analyzing scores and ranking information which focus on estimation of the "true" underlying qualities of cach item (Volkovs & Zemel, 2012; Shah ct al., 2016b; Hajek et al., 2014; Negahban et al., 2017; Wang et al., 2020) or the ranking of such assumed "true" qualities (Braverman & Mossel, 2007; Shah et al., 2016a; Shah & Wainwright, 2017; Pananjady et al., 2017; Volkovs & Zemel, 2012; Mao et al., 2017), we deliberately focus on obtaining a dequantized numerical output corresponding to each review. This is because, in the actual peer-review process, the final aggregation depends on various additional factors including the review and rebuttal text, reviewer discussions, and Program/ /rea Chairs' preferences how to aggregate individual reviews. The reviewer-provide initial judgments but do not fully determine the final decision. Furthermore, we deliberately do not want to base our algorithm on the assumption of the existence of some objective true qualities, especially in applications such as peer review that has a significant amount of subjective opinion (Cortes & Lawrence, 2021). Thus with our goal of helping the Program Chairs and Area Chairs seamlessly make their decisions in the real world, we focus on producing a unified representation in form of dequantized scores (r.p)( This approach of computing updated scores for each review is also taken by Noothigattu et al. (2021) in their work on mitigating subjectivity in peer review.
scores are usually used heavily in the peer-review workflow for sorting
Design Principle 1 (Dequantization of scores and not final decisions). The goal is to produce estimated dequantized scores (rpp)r and not final decisions on the set of papers.
There are two ways to interpret the dequantized scores (oppeeA one under model-based setting and another that does not rely on modeling assumptions.
(1) Under model-based setting, outcomes in form of partial rankings and scores generated from latent and real-valued variables. For example in the commonly-used Thurstone model (Thurstone, 1927), the latent variable Urp: (r,p) E A represents the inherent opinion of reviewer for paper p and is drawn from normal distribution. Under this modeling assumption, (rrp)rrp)) can be interpreted as estimations of ground-truth values {UrP)(. In other words, it is the scores that reviewers would have given without the quantization process, which reflect the "true" opinion of the reviewers. This model aligns with certain models proposed in the literature: the model considered in Scction 3.2 of Zarkoob et al. (2022) and model number 2 in Heyard et al. (2022) also assume real-valued latent score per reviewer-paper pair, that yields the quantized score.
(2) The assumption that reviewers generate some unquantized scores in their mind before giving (quantized) score, as assumed by models such as the Thurstone model, may not always hold in practice. As discussed earlier in Section 1, people may not be capable of evaluating items with fined-grained measure scales (Miller, 1956; Jones & Loe, 2013; Lietz, 2010; Shah et al., 2016b). In this case, one can alternatively consider the dequantized scores (rrp)r as a set of continuous values that integrate the information from quantized scores and rankings without assuming the existence of ground-trute mound-truth scores. The values of these variables are still compatible with the traditionally score-based decision rules, and other workflow elements used by Area and Program Chairs (Figure 1).
With this motivation, we now present our second principle which requires the estimates to be consistent with the imput. In this work, we assume that any integer 2 represents real values in the interval from 2 - 0.5 and z + 0.5, which we refer to as quantization interval of
6

---

Published in Transactions on Machine Learning Research (09/2022)
Design Principle 2 (Consistency).. The values of dequantization output (Drp (r.a)e (i) must be consistent with reniewwerprroedded quantized scores, that is, the dequantized value must be inside the quanlizalion interval associated with the quantized score, Zrp - 0.5 S Drp S 2rp + 0.5, for every (r,p) E A; and (ii) must also be consistent with aaviewer-provider rankings, that is, the ordering of (Trpla(rD)EA strictly respects Rr, for every reviewer r.
The above design principles leave us with many possible choices for Trp. (r.p) E A. For instance, consider a scores-only setting where we observe the quantized score žrp. For a reviewer-paper pair (r,p) E A, the quantized score Zr,p defines only the range possible values for the dequantized score Dr.p- Based on only the principles we have established so far, all output values Trp [Erp 0.5, 2rp + 0.5] are equally good. Consequently, we establish another design principle to break ties within this interval.
We use the consensus value among reviewers as the additional signal to break ties for quantized scores. The consensus value for a paper is defined as the average of the scores provided by the reviewers for that paper. Based on only the information from the quantizod scores given by other reviewers for that paper, it is natural to envisage that the dequantized score lies closer to the consensus value than away from it. In other words, the consensus signal indicates part of the interval [2r.p - 0.5, 2r.p 0.5] which the dequantized score should lie. For example, without any ranking information, if reviewer r gave a score of 7 and all the other reviewers gave 7 as well for paper p, then the consensus signal from other reviewers offers no additional information within the interval [6.5, 7.5]. However, if the other reviewers all gave quantized scores 4, then their consensus signal indicates that the dequantized score Dr. should lic within [4, n [7 - 0.5, 7 + 0.5]. which is [6.5, 7].
Design Principle 3 (Consensus). We break lães in scores due to quanlization in the direction of reviewer consensus.
How much does the dequantized score move in the direction of consensus? This is governed by hyperpa- rameter in our algorithm, whose value is chosen in a data-dependent manner.
4.2 Proposed algorithm
Our three design principles then lead to our proposed algorithm. Algorithm 1. The algorithm takes both reviewer-provided rankings and quantized scores as inputs, and outputs dequantized scores (rrpee (Design Principle 1). The algorithm solves constrained convex optimization problem. The two constraints ensure that the output values are consistent with the reviewer-provided information (Design Principle 2).
The term Urrp (Faa''p)) Er((r ()EA Vr'p) 2 in the objective measures disagreement between reviewers, therefore capturing the consensus (Design Principle 3). This term and the term (Urp- - = together capture the move towards the consensus, and the amount of movement is determined by hyperparameter X > O. In Section 4.4, we introduce proposed cross-validationn-ikk procedure called Quantization Validation for selecting X.
Algorithm Proposed algorithm
Inpuls: Quantized scores (#rp)(m))e and rankings (πr)re(R) given by reviewers, hyperparameter X, small
value e
Pre-processing:: Remove inconsistency म, = m M(r,p') (r, E A,
pp and Zpp < <zpp' J, Vr E [R]
Output: Solution of the following constrained convex optimization program:
2
arg min (ompere
Σ
Drp Hr' (r', .)) E AH
Ur'p M)E
Σ
+ Σ
(Arp zxp)?
pElP:
m(pple
(mp))d
such that Drp 2 Urp te whenever (r:p) E A, (r. A, and p xr
Zrp 0.5 S Vrp 2rp + 0.5 4 (r,p) A.
7

---

Published in Transactions on Machine Learning Research (09/2022)
It remains to specify the parameter c, which can simply be chosen to be small positive constant. This parameter is only meant to ensure that any partial ranking given by reviewer is strictly followed. We pick = 0.05 in our subsequent experiments while noting that our results are robust to the choice of small (Appendix A.4). Our algorithm is inspired in part by isotonic regression (Barlow et al., 1972): if c = 0, then the term (Vrp - =rp)² in the ovjective and the resulting constraints Drp 2 Dry' V (r,p) E A, (r.p') E A, simply represent isotonic regression to incorporate the rankings."
Pre- Pre-processing In practice, a reviewer might make mistakes when interacting with the interface and give rankings inconsistent with the scores. Practitioners can use a simple pre-processing step to resolve the inconsistency. The pre-processing step removes from the set of reviewer-provided rankings (Section 3) the pairwise rankings that are inconsistent with quantized scores: क, =r ((v,p') (r.p) E (r.1/) E A.p> p' and Zrp < 7rp') }.
The optimization problem. When À > O, the objective is strielly conver. Note that the convexity of the first term follows from the fact that this term is sum over functions, where cach function is a composite of a convex function and an affine funetion. The set of constraints are linear inequalities, therefore the optimization problem yields unique solution. Specifically, it is convex quadratic programming (QP) problem, and therefore can be solved in time polynomial in the number of reviewers and papers.
Using the outputs: Scores or percentiles. We can extract two kinds of information from the output of the algorithm to present to the Program Chairs and/or Area Chairs The dequantized scores for all reviewer- paper pairs in the assignment (0rp. (r.p) E A} themselves provide a convenient and easy-to-use interface that is much cleaner than showing the quantized scores together with raw rankings, as demonstrated in Figure 1. The scores produced by the proposed algorithm lie within the quantization interval associated with the quantized scores (Design Principle 2). Within the intervals, the scores are in arbitrary scale because determining their exact values would essentially require more stringent modeling assumptions than we make in this work. The use of arbitrary scales is not new and is employed in various other applications to express measurements in absence of absolute values (Adinolfi Sargent, 2017; Bucher et al., 2006; Kamat, 2019; Hoffman, 2004).
An alternative useful type of information that can be derived from this data is the ranking or percentile of each score Trp. which represents the relative position of evaluation given by to among the entire pool of all review scores (across all assigned reviewer- papper paper pairs). In Figure 1b, the percentiles can be shown in place of the dequantized scores and enjoy the same compatibility as dequantized scores with the workflow of the chairs. Either of these types of information can then be provided to an Area Chair or Program Chair to help inform decisions across a wider scope of papers that are not accessible to individual reviewers.
In the sequel, we primarily focus on evaluating the performance in terms of ranking error of the scores for all reviewer-paper pairs. The Kendall-tau ranking error we use is formally defined in Section 5.1.
4.3 Analysis of the proposed algorithm in special cases
In this section, we analyze several properties of Algorithm 1. First, we draw a connection to Balanced Rank Estimation (BRE) algorithm from Wauthier et al. (2013) (Section 4.3.1) under certain assumptions. We then characterize the output by giving its analytical solution when only scores are present (Section 4.3.2). We also make connection to the Thurstone model with quantization under the score-only case (Section 4.3.3).
4.3.1 Connection to Balanced Rank Estimation (BRE) when À
Balanced Rank Estimation (BRE) (Wauthicr et al., 2013, Scction 4.1) is an algorithm for rank recovery with pairwise comparisons which enjoys optimal theoretical guarantees under the standard comparison-only setting, under three assumptions: there a ground-truth global ranking, the pairs compared are chosen independently and uniformly at random, and each pairwise comparison has an identical noise distribution.
Note that isotonic regression itself cannot break ties between quantized scores for each reviewer.
8

---

Published in Transactions on Machine Learning Research (09/2022)
In nutshell, BRE first estimates score for each item. The estimated score is proportional to the difference between the number of items preceding and succeeding this item in the given pairwise comparisons. Then, BRE uses these estimated scores to induce a global ranking of all items and the global ranking is the final output of their method.
In what follows, we show the connection between our algorithm and BRE under certain setting. In this setting: (i) every reviewer gives a total ranking of all the papers assigned them, and (ii) == oo in our algorithm, that is, we retain only the second term in the objective in Algorithm 1. Without the first term in its objective, the optimization problem in Algorithm 1 can be solved separately and in closed-form for each reviewer, and simply corresponds to the problem of breaking ties amongst papers with the same score using comparisons. For a reviewer-paper pair (r,p) E A, the output Top equals the sum of quantized score Erp given by the reviewer and an arbitrary-scale value generated from the ranking ny. We next describe the connection between this ranking-induced value and the output of the BRE algorithm. Consider the set of pairwise comparisons indicated by the total ranking Tp, and further restrict our attention to the subset of comparisons between paper p and papers with the same quantized scores as p. The ranking-induced value is then proportional to the difference between the number of papers preceding and succeeding p in this subset of comparisons. This is the same way that the BRE algorithm estimates the score for paper p given the subset of comparisons as input.
The following proposition characterizes the behavior of the algorithm in the limit of À = DO.
Proposition 1. Assume that reviewers give total rankings of assigned popers, and that e is a small constant such that the program in Algoritlm is feasible. When = our algorithmm generates scores separately across different reviewers. For each reviewer, our algorithm adds the quantized scores values that are proportional to the estimated scores from the Balanced Rank Estimation (BRE) (Wauthier et al., 2013) algorithm.
Therefore, refer to the special case which our algorithm reduces to when X = oc and reviewers provide total rankings of assigned papers as BRE-adjjused--sorres We provide further details of this reduction and the full algorithmic description of BRE-adjustedd-coree (Algorithm 3) in Appendix B.1. This reduction is of interest since it shows that when x = oo, the proposed algorithm still makes reasonable use of the provided rankings to incorporate into quantized scores.
4.3.2 Analytical solution when only scores are available
While the solution of Algorithm 1 can be complicated to analyze, it is much easier when we only use scores. Without the ranking information, the only constraints on the output scores are Zrp - 0.5 S Yrp I 3rp + 0.5, V(r.p) p) E A. We can then get an analytical solution for Algorithm as below.
Proposition 2. Suppose reviewers only provide quantized scores and no rankings, and each poper receives p scores. Then for any reviewer-paper pair (r.p) the scores (rrp)A output by Algorithm con be written in closed form
ÿ 1+ dd
Σ
2rp +
M11 + A) 3r'p
p(1 + A)
T
if
ÿ E Izpp - 0.5, 2rp + 0.5]
Drp
2rp - 0.5
if ÿ < žrp 0.5
2rp + 0.5 if ÿ > 2rp + 0.5.
The full proof is provided in Appendix B.2. The output (rap)r is a convex combination of scores for paper p, where the weights depend on the constant μ as well as the hyperparameter X > 0. Preciscly, the relative weight on the quantized score given by reviewer r is 1 + μd whereas the weights on every other reviewer are 1 (note that the number of reviews per paper μ is typically a small constant). Assume that the scores of paper p are not the same across all reviewers. Then, as the hyperparameter À increases and the weight on consensus decreases, we can see from the reduction in Proposition 2 that the estimated score (ūep) approaches { one should expect. As a further special case, when the scores received by any
9

---

Published in Transactions on Machine Learning Research (09/2022)
paper p from all reviewers are identical to, say, Ep, the estimates Brp also reduce to 2p may expect. When À = x,, the put is simply Erp- This result proves reasonable behavior from the proposed algorithm when only score information is available.
4.3.3 Connection to Thurstone model with quantization when only scores are available
We now draw a connection between the consensus objective used in Algorithm 1 and the Thurstone model with quantization. In this section, we retain the assumption in Proposition 2 that only scores are provided and focus on the consensus objective. The Thurstone model (Thurstone, 1927) is a widely used statistical model. For example, is used for modeling peer grading in MOOCS in Piech et al. (2013). In the Thurstone model with quantization, scores are generated by the following process. Each paper is assumed to have an underlying true quality score zz. The latent evaluation score Yrp given by reviewer r to paper p is generated from normal distribution, whose mean is the underlying true score of paper p. That is, Vrp ~ NNa,, o²), for some value a that represents the standard deviation. Our setting involves quantization, the observed scores (=rp)( are quantized scores obtained by quantizing the latent scores {Uro (Mo)EA to integers, that is, 2rp = Lyrp (r,p) E A. We focus on analysis of variables (Brp) Since the quantization process from to z is itself deterministie, we consider the joint likelihood PP(z, x*) To maximize the likelihood, we take the maximization over both x",y. We focus on the solutions for y, which we aim connect our output to. The connection between the consensus objective (which uses scores) and the log-likelihood under the Thurstone model with quantization is made in the following proposition.
then assume
Proposition 3. The mazimizer (r,)) of the consensus objeclive has the largest likelihood under the Thustone model with quantization, that is,
Σ PE[P)
Σ
Σ Dr'r
arg min (Urp A
Urp
(Ir':('')) E AH
r:(r,p)EA
arg max ttrphtr.peed
max (xg)pe(P)
log (opp((za)) (Urp) (aphpe(p))
such that Zrp - 0.5 < Drp & Trp + 0.5 A (r.p)
The proof of this proposition is provided Appendix B.3. We thus see that our consensus principle (Design Principle 3) also follows we consider standard paramctric model.
4.4 Hyperparameter selection via Quantization Validation (QV)
In this section, we introduce a novel cross-validation-liko method to choose the appropriate value of hyper- parameter > Algorithm 1. The selected value of À is then given as an input to the algorithm. In hypothetical situation where we were to observe ground truth values Yrp for some pairs of (r.p) E A, it would be possible to perform traditional cross-validation where choose the value of X that achieves the best performance in estimation of the obscrved values of Vrp a holdout set. However, in practice, we only have access to the quantized scores. This motivates us to design a procedure to select X, which performs validation on a dataset constructed by further quantization of the observations. We call this procedure Quantization Validation (QV).
We present our QV procedure in Algorithm 2. In words, given set of possible values for X, we first construct validation set by further coarsening the observed quantized scores using fewer quantization bins. Then we select the best value of À that achieves the lowest loss in the recovery of the original quantized scores, via a function of our choice loss A x RIAI + R. We use Kendall-tau ranking error as the loss function, which is defined formally in the experiments Section 5.1. Specifically, we select a quantization function q R + R. to convert the quantizod score Zrp to 2/p,, V(r, E A that has fewer quantization levels. For example, in our subsequent experiments, we choose q(-) [ [ The new rankings are then re-computed from the quantized scores (rap) as ( We validate on the dataset consisting of {r(p)tra)) and { where the goal is to recover (rp))(m)) Given pre-specified set A of candidate values of X, compare the
3Tee notation [-] denotes the mapping from any real value the nearest integer.
10

---

Published in Transactions on Machine Learning Research (09/2022)
Algorithm 2 Quantization Validation (QV)
Require: hppuls: observations of quantized scores
rankings (Tr)re(R), set of possible values of the hyperparameter A, quantization function 9, loss function loss.
Calculate further-quantizer scores 4/p ((2p)), v(r,p) A.
2: Calculate rankings
3:
TH = {(ppp') Erp > Erp (r.p) E A, (r.p)
4: for X A do
5: Compute the validation error ex = 1oss(2, 2), where & is the vector of {Erph(mm)a and 2 of
(Zrp)(n))
end for
Oulpul selected value of hyperparameter X arg minxex ex. (Ties broken in favor of the smallest value.)
algorithm output (Erphtr-p) under each value of with its ground truth (r,pee and compute the ranking error. Finally, we select the value of À as the one that induces the smallest ranking crror in the validation process, where ties are broken in favor of the smallest value.
5 Experiments
We evaluate the empirical performance of our proposed algorithm on simulated data as well as on semi- synthetic dataset based on real-world data collected from the peer review process of the ICLR 2017 conference. The code for our algorithms and results is available online.
5.1 Implementation details
For each experiment, we run trials and plot the mean and standard error of the mean. We treat differences less than 10-4 in values of Vrp as ties.
Proposed algorithm. As discussed in Section 4.2, Algorithm solves a strictly convex optimization problem with linear inqqualities. We use CVXPY obtain the solutions: The solver we used is CVXOPT with the tolerance for feasibility conditions (feastol) set 10-6, The constant which enforces the strict inequality constraints is set as 0.05. Note that the algorithm is robust to the choice of (as long it is small, such that the problem in Algorithm 1 is feasible): we present experiments demonstrating its robustness to the choice of e in Appendix A.4.
Quantization Validation. We use exponential grid for candidate values of À in Quantization Valida- tion: A = (cxp(t/4) 0 40, E Z}. The quantization function is set to q()) = [2
Performance measure. We use the (normalized) Kendall-tau ranking error as the loss function. Intu- itively, given two rankings the same set of elements, the Kendall-tau ranking error measures the number of pairs of items whose relationships are reversed in the two rankings. More formally, we define to be the vector of (Urp)rmp) (n and ÿ to be the vector of (Trp) Then the normalized Kendall-tan ranking error between y and y is defined as the following.
Σ
1 T = 3rrr)) (T@.. x )+
[((r,)) (r', ./)) Vrp Ur' (ra)) ('
Note that the pairs ((:p), (r'. (')) which are tied in y are omitted from the computation above. In addition to the ranking error, also measure the fz-error between the output values (rrp)r and the ground truth values (Urp) as v Exprotaeed Wyr 1.
www.tauuuanaaaaaee scores
11

---

Published in Transactions on Machine Learning Research (09/2022)
0.6
3.09
Noraaoee Kendai 88U Linees
0.5
0.55
4.50
0.04
4.45
3.04
4.00
303
0.3
Bomaked
302
362
Rapossed a gorteen
BRF aty
Proponed styrttin BRE odjicted ccores Guantrrd scones
261
Procoeed
001
Prasonal (and
Quen and scanor
000
300
Number of papers por povvevie
Number of popers per revieee
(b) Varying number of papers per re- (c) Ez-error. Varying number of pa- viewer.
(a) Varying standrad deviation a.
pers per reviewer.
Experiment results synthetic data with varying levels of noise (standard deviation σ) and number of papers assigned to each reviewer.
Figure 2:
5.2 Baseline methods
We compare the dequantized scores output by our algorithm with the outputs of the following two natural bascline methods.
Quantized scores: We simply use the observed scores (rrp)( as the final output.
2. BRE--adjusted-scores (Algorithm 3): In Section 4.3, we have shown that this algorithm is equivalent to our proposed algorithm when À + oc and reviewers provide total rankings of assigned papers. In ICLR 2017 dataset, reviewers may provide partial rankings instead. We make simple adjustments to the baseline to accommodate, which are explained in Appendix C. At a high level, the partial rankings can be viewed as total rankings over (tied) groups of papers. Therefore, the adjustments to the baseline is that the output scores are adjusted from the quantized scores in groups instead of individually. For simplicity, refer to this baseline in the ICLR 2017 dataset BRF-adjusted- scores well, throughout this section.
5.3 Synthetic dataset
We evaluate the performance of the proposed algorithm when the data is generated from a Thurstone model. For P papers to evaluate, their true scores are drawn independently from the uniform distribution 2 Unif(1, 9] Vp E [P]. The latent unquantized score Yrp that is given by reviewer r paper p is drawn from the normal distribution: Urp (a o²), where the parameter a represents the standard deviation of the model, then clipped by [0, 10]. The quantized scores then generated by rounding (Urp)(n))e to the nearest integer such that 0 < 2rp < 10, Zrp E E, V(r,p) A. For simplicity, assign the same number of papers to each reviewer in the experiments. Similarly, we give the same number of reviewers (scores) to each paper. Given fixed numbers of papers per reviewer and reviewers per paper, the reviewer assignment A = {(r,p) whore r reviews p.r r E [R] E [P]} is generated uniformly at random from all possible assiguments. The default setting is set to P 60, a 0.5, each paper gets reviewers and each reviewer assigned 4 papers. We examine the performance of our proposed algorithm under various settings where we change the parameters individually.
Varying levels of noise. We obtain solutions with data generated from the Thurstone model with different values of the standard deviations and show the results in normalized Kendall-tau error in Figure 2a. Note that under the setting with very small noise, the performances of the two baselines are slightly worse than those under the setting with a larger noise. This is caused by a larger number ties between the quantized scores under a smaller noise level. Specifically, when a 1.0, the average percentage of tics in the output of the quantized scores and the BRE-adjusted-scores are 11.1% and 5.8% respectively. However, when o decreases to 0.1, the average percentage of ties in their output increases to 12.4% and 6.3%. By considering both
12

---

Published in Transactions on Machine Learning Research (09/2022)
consensus between reviewers and the provided rankings in dequantizing the scores, our proposed algorithm has less than 1.5% ties in its output and outperforms the baseline methods under all three settings. We further show the distribution of selected À over the 20 trials in Figure 6 in Appendix A. As the noise level increases, the distribution of selected by QV moves toward the larger end, which is an indication of decreasing weight on the consensus term in the objective in Algorithm
Varying loads. We also consider settings with varying numbers of assigned papers to each reviewer and varying numbers of reviews for each paper. While varying one of the two variables, we fix the other to be the same the default setting. The error rates with a varying number of papers assigned to each reviewer are shown in Figure 2b. As the number of assigned papers incrcases, the error of BRE-adjusted-score bascline decreases, similar to the proposed algorithm. This is because the simple comparison injection employed by the BRE-adjusteddssoree baseline lets the number of papers per reviewer directly dictate the number of possible values for dequantized scores. Results show that changes in the number of reviewers per paper do not affect the performance significantly so we defer the corresponding plot of error rates to Figure in Appendix A. In all settings with varying loads, our proposed algorithm consistently incurs smaller errors than the baselines. We present an additional result (Figure 2b) which is the performance of the proposed algorithm under the oracle X ("Proposed (oracle)"). The oracle X is the one that achieves the lowest error on the dataset, among the exponential grid for candidate values (Section 5.1). The small gap between the proposed algorithm with QV-selocted À and with the oracle X demonstrates the strong performance of QV. More results regarding the hyperparameter selection process are in Section 5.5.
Additional experiment settings. To further validate the effectiveness of the proposed algorithm, we also present results in the following experiment settings, per the suggestions of reviewers for our paper.
1. Varying number of papers (Appendix A.5). In practice, the number of submissions would often be larger, ranging from hundreds to thousands. We present experiments with larger mumber of papers ranging from 200 to 2000 and observe that the performances remain similar as the number increases.
2. Varying affinities between reviewers and papers (Appendix A.6). We consider the more general setting where each latent score Dr.p in the Thurstone model has different variance dr,p to simulate different underlying affinities between papers and reviewers (Stelmakh et al., 2019b). The proposed algorithm retains its advantage in this setting.
le error. In addition to the ranking error, we also report l errors on the synthetic dataset. The & errors are calculated with the same set of dequantized scores for which we report the ranking errors. Despite its primary focus on recovering the ranking among (r the proposed algorithm shows a small advantage in lg error as well, compared to the baselines. The & errors with varying numbers of papers per reviewer are displayed in Figure 2c. The results on the l error for varying the noise and loads are qualitatively similar in that our method is no worse and offers a small improvement. The plots are deferred to Figure 7 in Appendix A.3.
5.4 Semi-synthetic dataset based on real data from ICLR 2017
We conduct experiments on data from the peer-review process of the ICLR 2017 conference (Kang et al., 2018). There are P= 427 papers and every paper receives at least 3 reviews. For simplicity (so that number of papers is multiple of the reviewer load), we keep P p= 426 papers and retain 3 reviews for each paper by discarding some reviews (43 out of 1321 reviews are discarded). Since the reviewers are all anonymous in this datasct, we allocate papers to reviewers randomly (sampling from the entire pool of papers without replacement), subject to the constraint that a fixed number of papers are assigned to every reviewer. There are several other datasets presented by Kang et al. (2018), but those datasets are not suitable for our explanate purposes. In Appendix D, we provide explanations on why those datasets are not suitable.
Each review score comprises an integer from to 10, which we treat as the ground-truth values for un- quantized scores (Mr))() We generate the quantized scores by putting the original review scores in
13

---

Published in Transactions on Machine Learning Research (09/2022)
25
0.08
20
Normalizad Kondail-tau
0.08
Propased algorifim BRE-adjs226-aooe
Enrore
15
Quantized Proposed (orace)
0.04
Proposed algeritm
0.02
adjuted
Quanlowd seores
3
2
Number papers
Number of papers per reviewer
(b) lz-error. Varying number of papers per re- viewers.
(a) Varying number papers per reviewers.
Figure 3: Experiment results on ICLR 2017 data.
fewer quantization levels, via the quantization procedure: 2rp [upp/2], such that 1 S 2rp S 5, 2rp E Z, v(r, p) E A. The reviewer-provided partial rankings are generated from the original review scores as Tr = (pp.p)) Yrp > Brv's (r,p) E A. (r.p') E A}, since the ICLR 2017 conference did not collect rankings dircetly from reviewers.
Figure 3a plots the normalized Kendall-tau error of our algorithm and the baselines under varying numbers of papers per reviewer (note that cannot control the noise level in the ICLR 2017 dataset we do in the synthetic dataset.) We observe that our algorithm incurs a lower error compared to the baselines. Furthermore, the error decreases as the number of papers per reviewer increases. As this mumber decreases from 6 to 2, the percentage of ties in the output of quantized scores remains at 32.2%, while the percentage in the output of Bsst-ddjuste-sscooee increases from 8.0% to 23.2%. Our algorithm outputs less than 3.4% percent of ties in all three settings. Similar to Figure 2b, we plot the performance of the proposed algorithm under oracle X in Figure 3a ("Proposed (oracle)")) We observe that the performance of the proposed algorithm with oracle À (green line) have negligible difference with the proposed algorithm with QV-selected À (blue line). This shows the elfectiveness of QV.
Other than the random reviewer assignment, we also show results under "clustered" reviewer assignment, per suggestions of reviewers for our paper. In this sctting, each reviewer can only revicwer papers from the same cluster, in other words, subset all papers. This clustered structure sinulates different genres of paper, since the real assignment data not public. Details of these experiments are Appendix A.7.
Similar to the synthetic dataset, report the & errors the ICLR 2017 dataset in Figure 3b. Again, observe that our estimator slightly outperforms the baselines in terms of the lg error. More details can be found in Appendix A.3.
5.5 Hyperparameter À selection via QV
We present results that shed light on the Quantization Validation (QV) hyperparameter selection process introduced in Section 4.4, as well as the effect of hyperparameter X on the performance of the proposed algorithm. Figure 4 presents plots comparing the performance of different values of hyperparameter X, the value chosen by QV, and the best value of chosen by a hypothetical oracle that has access to ground truth data. The plots compare these choices on two synthetic datasets (with parameters set as defaults specified in Section 5.3) and the ICLR 2017 dataset (with 6 reviewers per paper). The errors are shown in log-scale for clarity for ICLR 2017 data, note that y-axes may not start at to be able zoom in the relevant parts. Our experiments (see also Figure 2b and Figure 3a) reveal strong performance of the Quantization Validation process: we observe that it can select good X close to the optimal idcal value X*, and incurs an error close to that incurred by X
14

---

Published in Transactions on Machine Learning Research (09/2022)
Normalized Kerdall- l-u eirore
Normalized Kardall tau etrors
0.030
0.030
Normalized Kardall-auu
10
0.025
0.025
0.020
0.016
0.015
10
0.010
10
10'
10"
10²
10
10'
10
10"
10"
10"
10
Values ofA
Values of
Values of
(a) Synthetic dataset 1: QV
(c) Synthetic dataset 2: QV
(e) ICLR 2017 dataset: QV
Nomellze Kondell-oan kiuu errore
0.046
Narmaizzd Kendall tau ertor
Nomellzed Kendall-su errors
Proposed algorithm
0.046
Propessed algonitmm
Proposed algorithm
0.044
Beseline: BRE-adjssed scores
Baseline: BRE- BRE-adjusted scores
2
Béseline: BRE-adussted scores
0.044
0.042
0.040
2x10
0.038
QV
0.040
QV
0.030
0V
0.036
0.036
0.031
10
10"
10'
10"
10*
10" Values of
Values o
Values of
(b) Synthelic dataset 1: original data (d) Synthelic dataset 2: original data (I) ICLR 2017 dalaset: original data
Figure 4: The ranking error with varying À in synthetic and ICLR datasets. Top row: Error on the quantization validation set. Red circles mark the value selected by QV which achieves the smallest errors on the quantization validation set. Bottom row: Error curves on the original dataset. The circles indicate performance under the hyperparameter value selected by QV (from the top row). This compared to the performance of optimal value marked by red diamonds, which achieves the smallest error on the original dataset. Note that the y-axis does not start at zero, in order to clearly depict the relative error of various hyperparameter choices.
6 Discussion
We address the problem of aggregating quantized scores and rankings evaluations in the form of dequan- tized scores, applicable to important scttings such as peer-reviow. We propose a computationally--fiiie algorithm that solves a coistr convex optimization problem to compule the dequantized scores. The proposed algorithm evaluated on both synthetic data and datasets based on real peer-review scores, and shows strong empirical advantages.
An aspect to keep in mind regarding any such adjustment that uses global data is that of privacy in peer review (Ding et al., 2020; Jecmen et al., 2020; Ding et al., 2022). By providing Area Chairs with the information aggregated across all the reviewers who reviewed their assigned papers, we need to ensure that it should not inadvertently reveal the review information of paper(s) that they are not handling. As illustrated Figure la, the Area Chairs only observe rankings between papers that they are handling. For example, an AC might see *A > B" where "*' is paper C whose information is hidden from the AC. In our algorithm, however, the dequantized scores of A and B might depend the score of paper C (and scores given by other reviewers who reviewed paper C, etc.). If A gets dequantized score of 6 while B gets 3.95, C might have tic with B with dequantized score of 4.05. Therefore, the dequantized scores may contain more information than the quantized scores plus the partially-obscured rankings revealed to the AC. Although it unlikely that the AC would be able to directly infer, for example, the identity of reviewers of paper C, is potential concern that some information about paper C would be revealed from the dequantized scores given the AC. Another direction of future work is that of global versus subgroup accuracy. For example, some subgroups of papers might have larger inter-reviewer disagreement, fewer
15

---

Published in Transactions on Machine Learning Research (09/2022)
wwiewwr-proovdder comparisons than other papers, because of their fields. This phenomenon might affect the ranking errors in these subgroups through the consensus objective in our algorithm. Therefore, it is also of interest to analyze and/or modify our proposed algorithm to ensure comparable error rates across various subgroups of papers, in addition to the global Kendall-tau ranking error that we currently study in this work. Finally, of interest to prove strong theoretical guaranlees about the proposed algorithm including our Quantization Validation method, or design new algorithms for this problem that have strong theoretical guarantees with good empirical performance.
7 Acknowledgements
This work was supported in parts by NSF CAREER award 1942124, NSF CIF 1763734, and a Google Research Scholar Award. We thank the AE Branislav Kveton and the anonymous reviewers for their valuable and prompt comments and suggestions on the manuscript.
Broader Impact Statement
The interest for an efficient and accurate peer review system has dramatically grown for modern large scientific conferences. This work presents a practical attempt using pairwise comparisons to improve the evaluation scores in conference peer-review and potentially, in other settings where quantized scores and comparison information are both available. As outlined in Section 6, practitioners should also be aware of the potential negative effects of our proposed method before applying il to a real setting, including the potential reveal of confidential review information and possible worse accuracy on certain subgroups of papers/reviewers. The quality of the updated scores from our algorithm might be affected by other problems in peer-review, such as malicious behaviors from paper authors. For example, a reviewer who is also a paper author might intentionally give low quantized scores or manipulate the rankings of assigned papers to improve the acceptance chance of their own paper. See Section for references to works that identify and propose solutions for those (parallel) problems, although it remains to connect them with our problem formulation. Evaluating these aspects forms useful directions for future work.
References
Valerio Adinolfi and Edward H Sargent. Photovoltage field-offect transistors. Nature, 542(7641):3244-222 2017.
Arpit: Agarwal, Prathamesh Patil, and Shivani Agarwal. Accelerated spectral ranking. In International Conference on Machine Learning, pp. 70 79. PMLR, 2018.
Nir Ailon. Aggregation of partial rankings, p-ratings and top-m lists. Algorithmica, (7(2):284-300) 2010.
RE Barlow, DJ Bartholomew, JM Bremner, and HD Brunk. The theory and application of isotonic regression, 1972.
Mark Braverman and Elchanan Mossel. Noisy sorting without resampling. arXiv preprint arXiw:070771051,, 2007.
Mark Braverman, Jieming Mao, and S Matthew Weinberg. Parallel algorithms for select and partition with noisy comparisons. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing, pp. 851-862, 2016.
Matthias Bucher, Antonios Bazigos, François Krummenacherr, Jean-Micehl Sallese, and Christian Enz. Ekv3. 0: An advanced charge based mos transistor model. design-oriented mos transistor compact model. In Transistor Level Modeling for Analog/RP F IC Design, pp. 67 95. Springer, 2006.
Corinna Cortes and Neil D Lawrence. Inconsistency in conference peer review: Revisiting the 2014 neurips experiment. arXiv preprinl ar xrXi::2109.09774, 2021.
16

---

Published in Transactions on Machine Learning Research (09/2022)
Komal Dhull, Steven Jecmen, Pravesh Kothari, and Nihar B Shah. Strategyproofing peer assessment via
partitioning: The price in terms of evaluators' expertise. arXiv preprinl arXiv:220 106331,, 2022.
Wenxin Ding, Nihar B. Shah, and Weina Wang. On the privacy-utility tradeoff peer-review data analysis. In AAAI Privacy-Presereing Artificial Intelligence (PPA1-21) workshop, 2020.
Wenxin Ding, Gautam Kamath, Weina Wang, and Nihar B. Shah. Calibration with privacy in peer review, arXiv 2201.11308, 2022.
John R Douceur. Paper rating vs. paper ranking, ACM SIGOPS Operating Systems Review, 43(2):117-121, 2009.
Brian Eriksson. Learning to top-k search using pairwise comparisons. Artificial Intelligence and Stalistios, pp. 265 273, 2013.
Norman T Feather. The measurement of values: Effects of different assessment procedures. Australian Journal of Psychology, 25(3):221-23) 1973.
Peter A. Flach, Sebastian Spiegler, Bruno Golénia, Simon Price, John Guiver, Ralf Herbrich, Thore Graepel, and Mohammed J. Zaki. Novel tools to streamline the conference review process: Experiences from SIGKDD'09, SIGKDD Explor. Newsl., 11(2):63 67, May 2010. ISSN 1931-0145. doi: 10.1145/1809400. 1809413. URL http::///o..ateeeeeo aetooi.acm.rrr. org//0//////0 http:/...11111111111111 1809413.
Bruce Hajek, Sewoong Oh, and Jiaming Xu. Mininax-optimal inference from partial rankings. arviv preprinl 0rXiv:146.55633 2014.
Rachel Heyard, Manuela Ott, Georgia Salanti, and Matthias Egger. Rethinking the funding line at the swiss national science foundation: Bayesian ranking and lottery. Statistics and Public Policy, (just-accepted): 1 27, 2022.
RL Hoffman. Zno-channel thin-film transistors: Channel mobility, Journal of Applied Physics, 95(10): 1813-5819, 2004.
Max Hopkins, Daniel M Kane, and Shachar Lovett. The power of comparisons for actively learning linear classifiers. arXiv preprint arXix:1907.0sx 2019.
Max Hopkins, Daniel Kane, Shachar Lovett, and Gaurav Mahajan. Noisc-tolerant, reliable active classifica- tion with comparison queries. In Conference Learning Theory, pp. 1957 2006. PMLR, 2020.
Steven Jecmen, Hanrui Zhang, Ryan Liu, Nihar B. Shah, Vincent Conitzer, and Fei Fang. Mitigating manipulation in peer review via randomized reviewer assignments. In NewrIPS, 2020.
W Paul Jones and Scott A Loe. Optimal number of questionnaire response categories: More may not be better. Sage Open, 3(2):215822411448800 2013.
Prashant V Kamat. Absolute, arbitrary, relative, normalized scale? how to get the scale right, 2019.
Dongycop Kang, Walced Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. A dataset of peer reviews (peerread): Collection, insights and NLP applications. Meeting of the North American Chapter of the Association for Computational Linguistics (NAACT), New Orleans, USA, June 2018. URL https://arxts httpaaaaw....arrararararaaaaaeee wrgg/b//////....o./ 09635.
Carole J Lee. Commensuration bias in peer review. Philosophy of Science, 82(5):1272 1283, 2015.
Petra Lietz. Research into questionnaire design: A summary of the literature. International journal of market research, 5242):249-272, 2010.
Michael Littman. Collusion rings threaten the integrity of computer science research. Communications of the ACM, 64(6):43 44, 2021.
17

---

Published in Transactions on Machine Learning Research (09/2022)
Rahul Makhijani and Johan Ugander. Parametric models for intransitivity in pairwise rankings. In The World Wide Web Conference, pp. 3056-30022 201 9.
Emaad Manzoor and Nihar B Shah. Uncovering latent. biases text: Method and application to peer review. In AAAI, 2021.
Cheng Mao, Jonathan Weed, and Philippe Rigollet. Minimax rates and efficient algorithms for noisy sorting, arXiv preprint arXiv: 10388, 2017.
George A Miller. The magical number seven, plus minus two: Some limits capacity for processing information. Psychological revieu, 63(2):81, 1956.
Sahand Negahban, Sewoong Oh, and Devavrat Shah. Rank contrality: Ranking from pairwise comparisons. Operations Research, 65(1):266 287, 2017.
Ritesh Noothigatlu, Nihar Shah, and Ariel Procaccia. Loss functions, axioms, and peer review. Journal of Artificial Intelligence Research, 2021.
Ashwin Pananjady, Cheng Mao, Vidya Muthukumar, Martin J Wainwright, and Thomas A Courtade, Worst-case average-case design for estimation from fixed pairwise comparisons. arXiv preprint arXiv:1707.06217, 2017.
Michael Pearce and Elena A Erosheva. A unified statistical learning model for rankings and scores with application to grant panel review. arXiv preprint XrXiv:2201. 022539,, 2022.
Chris Piech, Jonathan Huang, Zhenghao Chen, Chuong Do, Andrew Ng, and Daphne Koller. Tuned models of peer assessment in moocs. arXiv preprint arXiv:1307.290 2013.
William Rankin and Joel W Grube. A comparison of ranking and rating procedures for value system measurement. European Journal of Social Psychology, 10(3):233-244 246, 1980.
Anna Rogers and Isabelle Augenstein. What can we do to improve peer review NLP? arXiv preprint arXiv:2010.00 03863, 2020.
Magnus Roos, Jörg Rothe, and Björn Scheuermann. How to calibrate the scores of biased reviewers by quadratic programming. In AAAI Conference Artificial Intelligence, 2011.
Nihar Shah, Sivaraman Balakrishnan, Aditya Guntuboyina, and Martin Wainwright. Stochastically transitive models for pairwise comparisons: Statistical and computational issues. In International Conference on Machine Learning, pp. 11-20, 2016a.
Nihar B Shah. An overview of challenges, experiments, and computational solutions in peer review. Com- munications the ACM. Extended version available at
2022.
Nihar B Shah and Martin J Wainwright. Simple, robust and optimal ranking from pairwise comparisons.
The Journal of Machine Learning Research, 18(1):7246 7283, 2017.
Nihar B Shah, Joseph K Bradley, Abhay Parekh, Martin Wainwright, and Kannan Ramchandran. A case for ordinal peer-evaluation in mooes. In NIPS Workshop on Data Driven Education, pp. 1-8, 2013.
Nihar B Shah, Sivaraman Balakrishnan, Joseph Bradley, Abhay Parekh, Kannan Ramchandran, and Mar- tin J Wainwright. Estimation from pairwise comparisons: Sharp minimax bounds with topology depen- dence. The Journal of Machine Learning Research, 17(1):2049 2095, 2016b.
Nihar B Shah, Behzad Tabibian, Krikamol Muandet, Isabelle Guyon, and Ulrike Von Luxburg. Design and analysis of the nips 2016 review process. The Journal of Machine Learning Research, 19(1):193311966, 2018.
David Socrgel, Adam Saunders, and Andrew McCallum. Open scholarship and peer review: a time for experimentation. 2013.
18

---

Published in Transactions on Machine Learning Research (09/2022)
Thane Somers, Nicholas RJ Lawrance, and Geoffrey A Hollinger. Efficient learning of trajectory preferences using combined ratings and rankings. In Proc. Robolics: Science and Systems Conference Workshop Mathematical Models, Algorithems, and Human- Robot Interaction (RSS), Boston, MA, 2017.
Ivan Stelmakh, Nihar Shah, and Aarti Singh. On testing for biases in peer review. Advances in Neural Information Processing Systems, 1452886-529,, 2019a.
Ivan Stelmakh, Nihar B Shah, and Aarti Singh. Peerreviewdall: Fair and accurale reviewer assignment in peer review. In Algorithmic Learning Theory, pp. 828 856. PMLR, 2019b.
Sijun Tan, Jibang Wu, Xiaohui Bei, and Haifeng Xu. Least square calibration for peer reviews. Advances in Newral Information Processing Systems, 34, 2021.
Louis Thurstone. law of comparative judgment. Psychological review, 34(4):273, 1927.
Andrew Tomkins, Min Zhang, and William D Heavlin. Reviewer bias in single double-blind peer review. Proceedings of the National Academy of Sciences, 114(48):12708-222 120011221181813 2017.
T. N. Vijaykumar. Potential organized frand in ACM/IEEE computer architecture con- ferences. http:///////aaaaooo/////oooootto cow.rtureeecoooeooc--ee-eee--ooo
fraud- ian-in--cm -eee- computer-
2020.
Maksims N Volkovs and Richard Zemel. flexible generative model for preference aggregation. Proceedings of the 21sl international conference on World Wide Web, pp. 479 009,488 2012.
Jingyan Wang and Nihar B Shah. Your 2 is my 1, your 3 is my 9: Handling arbitrary miscalibrations in ratings. In AAMAS, 2019.
Jingyan Wang, Nihar Shah, and R Ravi. Stretching the effectiveness mle from accuracy to bias for pairwise comparisons. In International Conference on Artificial Intelligence and Statistics, pp. 66 76. PMLR, 2020.
Fabian Wauthier, Michael Jordan, and Nebojsa Jojic. Efficient ranking from pairwise comparisons. In International Conference on Machine Learning, pp. 109 109-117, 2013.
Ruihan Wu, Chuan Guo, Felix Wu, Rahul Kidambi, Laurens van der Maaten, and Kilian Q Weinberger. Making paper reviewing robust to bid manipulation attacks. arXie preprint arXiv:2102.0600 2021.
Yichong Xu, Hongyang Zhang, Kyle Miller, Aarti Singh, and Artur Dubrawski. Noise-tolerant interactive learning using pairwise comparisons. In Neural Information Processing Systems, NIPS, 2017.
Yichong Xu, Han Zhao, Xiaofei Shi, and Nihar Shah. On strategyproof conference review. In IJCAI, 2019.
Yichong Xu, Sivaraman Balakrishnan, Arthur Dubrawski, and Aarti Singh. Regression with comparisons: Escaping the curse of dimensionality with ordinal information. Journal of machine learning research, 2020a.
Yichong Xu, Xi Chen, Aarti Singh, and Artur Dubrawski. Thresholding bandit problem with both ducls and pulls. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics, AISTATS, 2020b.
Yichong Xu, Aparna Joshi, Aarti Singh, and Artur Dubrawski. Zeroth order non-convex optimization with dueling- choice bandits. In Conference on Uncertainty in Artificial Intelligence, UAI, 2020c.
Hedayat Zarkoob, Greg d d'Eon, Lena Podina, and Kevin Loyton-Brown. Brown. Better peer grading through baycsian inference. arXiv preprint arXiv:2200.012 01242, 2022.
Shiwei Zeng and Jie Shen. Learning halfspaces with pairwise comparisons: Breaking the barriers of query complexity via crowd wisdom. arXiv preprint arXim:2011.01100 2020.
19

---

Published in Transactions on Machine Learning Research (09/2022)
A Additional experimental results
In this appendix, we present additional experimental results supplementing those in Section
A.1 Performance with varying numbers of reviewers per paper
In Figure 5, we show ranking error rates on the synthetic dataset with varying numbers of reviewers per paper. Change of this number does not affect the algorithm's performance significantly.
A.2 Distribution of hyperparameters selected by QV
In Figure 6, show the distribution of hyperparameter value chosen by the QV procedure (Algorithm 2) synthetic datasets. The distribution of values changes most significantly the noise standard deviation increases. This is because increasing a makes scores more divergent for each paper. The QV effectively captures this and as result, chooses larger À as the noise level increases, which leads to decreasing weight on the consensus term in the objective of the optimization problem in Algorithm 1. The change in the number of papers per reviewer, or the number of reviewers per paper does not significantly affect the range of hyperparameter values selected by the QV. This indicates that these two parameters have no significant effect on how much the algorithm relies on consensus, as opposed the noise level o.
A.3 Performance in & errors
We display the additional results in & error synthetic data in Figure 7. For the ICLR 2017 dataset, where the l errors are shown in Figure 3b, provide additional details as follows. Given the data- generation process, we first project the dequantized scores back to integers in [1, 10] by the function Trp = 22r.p - 0.51, (r.p) E A. For example, Trp in the interval [1, is projected to 2, and Drp in the interval [1.5.2] 2) is projected to 3. For synthetic dataset, we provide the additional results of performance in le error with varying levels of noise and numbers of reviews per paper. The results under these settings are shown in Figure 7.
A.4 Performance with varying e
We show that the proposed algorithm is robust to several choices of small We vary the value of for our proposed algorithm and obtained results both synthetic and ICLR 2017 data, which are shown in Figure 8, For synthetic data, the parameters are set to default (Section 5.3). For the ICLR 2017 data, the only parameter which is the number of papers per reviewer is set to 6. Performances of the baselines are also plotted as a reference.
0.08
Nomellzd Kencallaa Karccalaa Erros
0.05
0.04
0.03
0.02
Proposnd aigorthm
3.01
BRE adjusted--ccees
Quantzzd scores
2
4 Number of revieierrs per
Figure 5: Synthetic data with varying numbers of reviewers per paper.
20

---

Published in Transactions on Machine Learning Research (09/2022)
1.0
1.0
0.8
0.8
Frequency
0.6
Frequencr
0.6
0.4
0.2
0.2
0.0
0.0
2000
4000
6000
2000
4000
6000
Values of lambdas
Values of lambdas
Noise level 0..1
(b) Noise level 1.0
1.0
1.0
0.8
0.8
Frequency
Frequency
0.6
0.4
0.2
0.2
0.0
0.0
500
1000
1500
2000
500
1000
1500
2000
Values lambdas
Values of lambdas
(c) Number of assigned papers per reviewer
(d) Number assigned papers per reviewer
1.0
1.0
0.8
0.8
Frequency
Frequency
0.6
0.4
0.2
0.2
0.0
0.0
500
1000
1500
2000
500
1000
1500
2000
Values lambdas
Values of lambdas
(e) Number scores per paper
(f) Number of scores per paper
Figure Distributions of selected by QV in synthetic dataset with varying parameters.
21

---

Published in Transactions on Machine Learning Research (09/2022)
zoom In VIWW
zoom view
4.5
Errore
Emrrrs
4.4
0.25
0.5
0.7
1.0
Proposed algotthen
Praposed aigorihmm
BRF-adj adad-acoria
Uuantzed scores
Quanlized scvess
0.75
1.0
3
The standard deviation parameter
Number of revizwers per
(a) Varying noise level a.
(b) Varying number reviewers per paper.
Figure 7: Additional experiment results on le errors on synthetic data.
0.05
0.05
Normllzed Kerdal-iuu Erooe
Kardal-iau
0.04
Proposed algorithm
0.03
adjusted
0.34
Quentizd acores
0.02
Proposed algonthm
0.02
0.11
BRE-aajannddaccore
Quantized
0.00
0.00
0.05
0.025
aa) Experimental results synthetic data.
(b) Experimental results ICLR 2017 data.
Figure 8: Experimental results with varying c.
A.5 Performance with large number of papers
To validate the performance of the proposed algorithm on datasets with large numbers of papers, we conduct additional experiments with synthetic data, increasing the total amount of papers 2000. The other parameters are set to default (Section 5.3). The results are plotted in Figure 9. Similar to the numbers of reviewer per paper (Figure 5)), the total number of papers does not have significant impact on the
Nomalizd Kancal-au Errors
0.04
0.03
0.32
Propoond aigorthm adjusted-eccres
0.01
Quantzed scores
200
500
1000
2000
Number
Figure 9: Experiment results Synthetic data with large number of papers.
22

---

Published in Transactions on Machine Learning Research (09/2022)
KendiladdKnnddliee Eroos
0.05
Normalizad KendalHHal
0.04
0.03
3:12
Proposed algerihm
BRE adjusted
uniized soves
0.00
Number papers reviewer
Figure 10: Experiment results Synthetic data with variances from uniform distribution.
0.08
0.8
Normllzzdd Kendail-tu Brors
Kancallau
0.08
0.06
Proposed algorithm
Proposed algerithm adjusied Quantized acores
BRE axjuseddeeeeeee ecores
Quantized
0.04
0.02
0.00
0.00
Number of papers per reviewer
Number papers per reviewer
(b) Uneven cluster
sizes in
proportion
(a) Similar cluster sizes.
4 2 1:1.
Figure 11: Experiment results on ICLR 2017 data with clustered reviewer assignment.
normalized Kendall-tas tau ranking error (Section 5.1). The results show that the proposed algorithm continues to outperforms baselines as the total number of papers grows large.
A.6 Performance with varying affinities between reviewers and papers
In the Thurstone model (Section 4.3.3), the variance o² in the normal distribution of the latent, score Yr.p be viewed as presentation of the underlying affinity between paper p and reviewer (Stelmakh et al., 2019b), in other words, the reviewer's expertise on their assigned paper. In our synthetic experiments (Section 5.3), the variance o² is set to be the same for all reviewer-pape pairs. In this subsection, we let Dr.p ~ (( o for each (r, p) in the assignment A, dr.p drawn independently and identically from a uniform distribution (0.1,, 0.9]. We plot the results under several choices of numbers of assigned papers per reviewer in Figure Our proposed algorithm remains dominant the performance various seltings.
A.7 Performance with clustered reviewer assignment
The reviewer assignment in the ICLR 2017 dataset is generated randomly (Section 5.4) because the real reviewer assignment data is not public. In this subscction, we experiment with the following "clustered" reviewer assignments as a middle ground. Note that since the review scores in ICLR 2017 dataset are given, we cannot vary the reviewer-pane paper affinities the same way as in the Thurstone model in Appendix A.6. We divide the papers in the ICLR 2017 dataset randomly into several clusters, and a reviewer can only review papers in the same cluster. The clustered structure simulates different genres of papers, for example, the Theory of MT. Deep learning, etc. We choose the number of clusters to be 4 and considered two options for
23

---

end for end for Published in Transactions on Machine Learning Research (09/2022)
cluster sizes: (i) all clusters are of approximately the same sizes and (ii) cluster sizes uneven. Within each cluster, we assign papers to reviewers randomly. We plot the results for these two cluster sizes in Figure 11. We observe that under the implemented clustered assignments, the performances remains similar to those in Figure 3a.
B Proofs of Propositions
In this section, we present the proofs for propositions from Section 4.3.
B.1 Proof of Proposition
For clarity, we first present the full procedure of BRE-adjnstcddscores here. is also one of the bascline methods that we evaluate Section 5, for syuthetic data and with a simple adjustment for the ICLR 2017 data. For conciseness, we fix the number of assigned papers across reviewers and denote this constant K.
Algorithm 3 BREBaadjusted-scores
Require: The ranking and quantized scores provided by each reviewer: (Zrph(r,)) and { {Te)re( For simplicity, assume ग,, is total ranking among papers TT = {px > Px- > .... p1). A small constant. e.
1: for [reviewers] do
2: Divide all the reviewed papers by their quantized scores, denote the set of quantization bins as for B E B do Define the ranked papers in B as (Pu Pu-1 > po), where and + represents
number of papers in this quantization bin. Denote the score value which is the same for all papers in B as 2g.
5:
for t u..... do Sct Tr.p. 2B + (t v) x..
6:
7: end for
Adjust the values Dr.p Ur.p (1 E Fr - 2B) for p E B.
10:
11: Output (Trp)
We first introduce the notion of quantization bins, which is defined separately across different reviewers. For reviewer r, quantization bin is group of papers with the same quantized scores given by Each reviewer can have at least 1 quantization bin and at most к quantization bins in the assigned papers. In Algorithm 3, the quantized score 2rp is incremented by a value according to the rank of p inside its quantization bin, then "centered" to form the output whose mean value inside each quantization bin remains the same. Precisely, Trp can be explicitly written as follows. Note that in Algorithm 3, the choice of does not change the ranking of output (rapeA and for implementation, we choose be the same as in the proposed algorithm, which is 0.05.
1 Prp Trp = 2rp + (r(p)
(1)
2 2
where Prp denotes the total number of papers in the bin that p belongs to, within the scope of reviewer and Tz( denotes the ranking of inside its quantization bin and within the scope of papers reviewed by r. For example, if reviewer r reviews only two papers and p', and gives identical scores to the two papers, and ranks them as > p/, then we have (.- = 2, xp/p =.
Having established the procedure in Algorithm 3, we now study (i) the reduction from our proposed algorithm \ = to BRF-adjusted--oomm and (ii) the connection between BRF-adjjuteei-scores and BRF.
24

---

Published in Transactions on Machine Learning Research (09/2022)
B.1.1 Reduction from proposed algorithm to BRE-adjusted-seee
Recall that, as À = oo, the proposed algorithm reduces to the following optimization problem which does not have the objective term that captures reviewer consensus. Ør is the vector of (Trp)(ra)e and yrs 2p are that of (yrp)trppe (raje(
Dr = arg minlly, 2r112,
(2)
9.ER.
such that Urp 2 Yry te whenever (r,p) E A, (r,p'))) A, and p > 2rp 0.5 S Uro 2rp + 0.5 A (r,p) E.A
(3)
Recall that we assume e to a small constant such that the feasible set is not empty. For example, in practice we choose be 0.05 (Section 5). Given the problem in (2), the solutions Drp are different from Trp only when there exist tied scores given by a reviewer. Precisely, with the assumption that reviewers report. total rankings, consider the problem of finding the vector
[x] ... 2m E Tn such that æ1 22 = Im (m 1)e that minimizes the objective function ||x clI [c, c.....d cl E R where is some constant. Let us set the derivative the objective function with respect to vm to find the minimizer:
17
fm arg min Σ
(x + ie c)² + Im
Therefore ã; = 2m + (m - i)E = c ((m i) e. In the setting of (2), corresponds to the vector of scores of papers with tied scores for fixed reviewer. is the score value, m is the total number of papers in the quantization bin. Therefore, we have the solution to the problem in (2) as:
Zrp + (T)())- - -1 Prp
Trp 2rp + 6 (m -
1
Pre
2rp +
(p)
2
2
It is casy to see the equivalence between (1) and (4), indicating that under the assumption that each reviewer provides lotal ranking क, among all assigned papers, the proposed algorithm reduces Algorithm A 0C.
B.1.2 Relationship between BRE-adjusted- -coress and BRE
Given possibly noisy, randomly collected pairwise comparisons where each pair is compared with a certain probability, the BRE estimates a score for each item (paper). The final goal in their work is global ranking, which is revealed by the estimated scores. In their setting, each pair can only be compared once, and the score of an item is caleulated as the relative number of items preceding and succeeding it. Using notations, let us denote the output score of paper p x* p: since they do not distinguish between different, reviewers. In BRE algorithm, the estimated scores for paper is
[P] pppp' -1 [P] P.P' pl.
(5)
Recall the adjustment to scores performed by BRE-adjusted-scores in (1). For a pair of reviewer and paper (r.p), let B, (p) denote the set of papers reviewed by and are in the same quantization bin as We can write that:
Prp = 1/' P (p), < Tp) = lo' : B, +1
of p' e B. > 11,
Plugging the above back in (1) gives us:
Drp = 2pp + (1p' :p pp (,),,p'' < 1p' : (,)))
(6)
25

---

Published in Transactions on Machine Learning Research (09/2022)
It is easy to that the adjustment amount conditioned on the quantized score in (6) is proportional to the relative difference between the number of papers preceding and succeeding paper p in its quantization bin. The multiplicative factor is simply the small constant e/2, since the output scores are arbitrary-scale (Section 4.2).
Combining B.1.1 and B.1.2, we proved the following the special case where (i) reviewers report total rankings of assigned paper and (ii) À oo in Algorithm 1: Algorithm adds each quantized score a value that is proportional to the estimated score from BRE, when BRE is given the ranking information within the quantization bin.
B.2 Proof of Proposition 2
Recall that μ is the number of scores received by each paper. Without ranking information, (Trp) can be solved separately for cach paper p [P]. For a fixed p, the proposed algorithm reduces to the following optimization problem.
arg min (epae
Σ reviewers r((p))Ed
(Urp vp)2
A
(Urp 2xp)²
(7)
(mp)Ed
such that 2rp 0.5 S Vrp Erp + 0.5 A (r,p) E.A
When À > O, the objective strictly multivariate convex function. For every reviewer that reviews paper p, the partial derivative of objective in (7) is as follows. For simplicity, let us denote the objective in (7) fp.
OSp = Dyrp
Vrp
Σ rr(ppeA
Dry + A(Urp Zep)
P
When Drp = x(1*X) 3rp + Eeerr 1114x]22'p 2r'p- the convex multivariate objective function f, achieves local minimum since its derivatives achieve 0 for all variables. Since the function is strictly convex, the local minimum is the global minimum. If the global minimum is inside the feasible set defined by the linear incqualitics, then the solution to the problem in (7) is
1+#d
T.P = M(1 + A) 2rp + Σ r'gr
1 as
m(1+ 2r'p.
(8)
In other words, [Urp) is the weighted average of scores for paper p, where the weights are dependent on the global minimum is outside
constant μ (number of reviews for each paper) and the hyperparameter the feasible set, observe that the objective function in (7) convex each variable yrp if other variables are fixed. Therefore, in this case, the solution (Ürp) } is the closest point in (Zyp 0.5, zpp + 0.5) to the right-hand side in (8). To summarize, the solution to (7) is follows.
Drp min max
((1+
1 nA
2rp Σ Mar
p(1+
2rp 2
1
2r'p:
0.5
2rp + 0.5
(9)
B.3 Proof of Proposition
The likelihood for all r.p: (r.p) A which we study can be expressed as
P ((zpp) (ump)(=;)) x P ((zrp)) (Urp) (x P ((um)) {xx)))
We thus have the following equivalence for taking maximization over the latent ys and x's. Note that we study the dequantized scores, so locusing the solutions for Dr.p.
arg max m(2rp( {Urp} xx)) arg max max max (Brp), P(yyrpt (x+))
(10)
(1.2} (zz)
(
{2z)
26

---

Published in Transactions on Machine Learning Research (09/2022)
Given observations of (r.))) the likelihood P ((2rp (r.)) (r.p)) (pp)re(rt) equals only when such consistency is satisfied: Drp E (rpp 0.5, 2rp + 0.5), ((r,)) E A and equals 0 otherwise, due to the deterministic nature of the quantization process. If consistency is satisfied, we then consider optimization of the second likelihood on the right-hand side by taking the logarithm of it.
(Urp xpp
log P((vp) xg))
Σ
log 0V2T
20²
ri(r,))A
Σ
(yrp xxpp?
(11)
r:(pp)E
For a fixed paper p, we can find the maximizer I (11) by setting the derivative 0 which yields x p [r'rr En(ra)a Urp for all p E [P]. Plugging xt p: vp E (P) back into (10) and we have that:
arg max max log P(grpp) (Mmp)aa)) arg min (drp)
E(yrp 1( (r',) AH
1
Σ Mrp)?. <((D)EE
[Urp) (a5)
s.t. 2rp 0.5 S Vrp S 2rp 0.5
Therefore, the maximizers (UrpTr.p of likelihood function under the Thurstone model with quantization is the same as the maximizer of the consensus objective Algorithm 1, with the constraints that Yro are consistent with the quantized scores 2rp's.
C Baseline method for ICLR 2017 data
In Algorithm Section B.1, we show the baseline method when reviewers provide total rankings of assigned papers. However, in the ICLR dataset (Section 5.4), we derive the reviewer-reporter rankings from their original review scores, since ranking information are integers in 1 ~ 10, and the rankings are derived as: = {(p,'' Mrp > 'rr' (r,p) A, (r.p/) A). Therefore, the reviewer-reported rankings are not total rankings over assigned papers, whenever there exist tics in the original review scores (Urp (r Instcad, they can be seen as a total ranking over groups of paper. Precisely, the rankings observed in this dataset are a subset partial rankings that can be expressed as n, (9T > 9T-1 > ... 91), where each G;, i [T] represents a group of paper with tied scores, p > p' if p E G., p' E Gy and u, a > U, [7], E [T]. For input to algorithms, if Yrp Vry', then p.p' E G. for some i E [T].
not collected directly from reviewers. The original scores
For the ICLR 2017 dataset with partial rankings, we employ baseline algorithm that can be seen generalization of Algorithm 3. The baseline method for the ICLR 2017 dataset is defined in Algorithm 4. The difference is that output scores are now adjusted from the quantized scores in groups instead individually. Precisely, the score adjustment of an item is not dependent on the number of items preceding and succeeding it, but on the number of groups precoding and succeeding group. When reviewers provide total rankings of assigned papers, each group only contains one paper and consequently, Algorithm 4 reduces to Algorithm 3.
In Algorithm 4, the value of remains 0.05.
D Limitations of the other datasets in Kang et al. (2018)
Kang et al. (2018) present several other datasets besides the one collected from the ICLR 2017 conference. These datasets are 1. arXiv 2007-2017; 2. NeurIPS 2013-2017; 3. ACL 2017: 4. CoNLL 2016. Below, we clarify one by one why these datasets are not suitable for the purposed of this paper.
Xiviv There are no review scores available for arXiv submissions.
NeurIPS Only review texts and the confidence scores for accepted papers are available. Furthermore, no review scores are available.
27

---

Published in Transactions on Machine Learning Research (09/2022)
Algorithm 4 Partäal-aukknnsssdllsseessorr
Require: The ranking and scores provided by each reviewer: (Arp) and (ā>)e((n) By is partial ranking among papers = (9T > GT-1 ....... A small constant
1: for E [reviewers} do
2: Divide all the reviewed papers by their quantized scores, denote the set of quantization bins as B. 3: for B E B do
if papers (P) belong more than one group then
Define the set of groups in as {9, Gu-1 >.... Go), where u. 2 v and u-u-1 represents the number of groups in this quantization bin. Denote the score value for bin B as 2g. for ==
do
7: end for
Set Dr 2B + (t - 1) x e, for all E Gt.
8:
Adjust the values Tr.p = Fr.p"
(RSDEB EMDES Dr BB for E B.
10: end if
11: end for
12:
end for
13:
Output ((rp)rMP)
ACL and CoNLL The original scores in these two datasets
scale of to 5. The proposed algorithm (Algorithm 1) runs on data with 5 levels of quantization and works well on such data, supported by our results on the ICLR 2017 data in which the observed scores have only levels. The only caveat is that, if one additionally wants to evaluate the algorithm's performance by using the original scores as ground truth as we did on the ICLR 2017 data, they would not able to do so on the ACL the CoNLL data, due to the small number of levels their original scores.
28


Reviews:
[{"review_id": "6CLmPIKIgq", "reviewer": "Reviewer_Dx6m", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The investigated issue is interesting and important for the peer review process. Ranking information is indeed not well-leveraged. Combining rankings and quantized scores should generate more reliable review results.\n- They formulated the problem mathematically and converted it into a convex optimization problem. They also analyzed the special cases.\n- They conducted experiments on real-world datasets and demonstrated the effectiveness of the proposed approach. Detailed analyses are given to study the impact of hyperparameter selection.\n\n### Weaknesses\n- Nowadays, we usually have thousands of submissions to machine learning related conferences each year. The tested datasets are too small. Also, they only tested the algorithm in the machine learning field. It would be more interesting if experiments on more domains were explored. As the authors mentioned in the broader impact statement, there are many other problems that may affect the quality of the updated scores.  To ensure robustness, more experiments are expected. \n- I am also concerned about the fairness of the algorithm. It would be better if there were some analysis on that.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to integrate the ranking information into the scores in the peer review process, with the goal of mitigating the arbitrariness in conference reviews.  They evaluate the proposed approach with synthetic datasets and the ICLR 2017 conferences, and it shows that the proposed approach can largely reduce errors (30%).", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- Adding more experiments with larger datasets.\n- Adding experiments on other domains.\n\n", "broader_impact_concerns": "This algorithm might be related to the fairness of peer review.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XTdTWUarcj", "reviewer": "Reviewer_Bb4m", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S1: The paper is very well written, easy to follow and covers the important issues well. The visualizations are carefully plotted to emphasize key information, e.g. the \"zoom-in\" in Figure 2 (c). The Reviewer really appreciates the efforts.\n\nS2: The paper studied a very practical problem in paper reviews. Examples given are intuitive and convincing. However, The Reviewer hasn't served as an AC yet, so the assessment of the problem is based on intuition.\n\nS3: Good analysis. The paper compared the proposed algorithm with reasonable baselines (BRE/Trunstonian) both in theoretical analysis and experiments. The paper also discussed the convex nature and the complexity of the solution, which helps avoid randomness.\nS4: Experiment setup. It's a hard problem to evaluate something without ground truth, but the paper did a reasonably good job to provide useful data.\n\nW1:  The goal is less clearly defined. On one handl, the task lacks true labels, and the authors deliberately said they won't assume the existence of true labels. On the other hand, the goal is defined vaguely as \"seamlessly assist the human experts\" (Section 3). It's not clear what a good result really means. \n\nFor other minor questions, please see Requested Changes.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed an algorithm that could help Area Chair (AC) to make better decisions about the acceptance of research papers (referred to as The Algorithm here after). It turns the quantized score and/or partial ordering of papers within each reviewer into a single dequantized score. The resulting score is easier to understand in AC's workflow, since they don't have to consider both a score and partial ordering.\n\nThe authors designed the algorithm to respect reviewer consent, be consistent with known partial ordering, and keep the resulting score as close to the original score as possible. Those principles are reflected in the design of the loss function. Finding the best score is a convex optimization problem where a single best solution is guaranteed and computationally tractable.\n\nThe authors compared the proposed algorithm with several lines of previous work, notably with BRE and Trunstonian models. Analysis showed that under certain constraints it agrees with the two reference models.\n\nTo estimate the hyperparameter lambda without labeled data, the authors proposed a Quantization Validation method, which takes current quantized scores as the truth, and quantizes them more to create a validation set. \n\nThe authors used a few synthetic data sets and ICLR 2017 data to evaluate the proposed algorithm with Kendall-tau ranking error. The results showed that the proposed algorithm performs better than the baseline (BRE-adjusted) under various factors (number of reviewers per paper etc). It outperforms baseline on ICLR data too, although by a smaller margin. Further evaluation of Quantization Validation showed that it can find the near optimal hyperparameter.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1) Page 4 Paragraph 3: The Reviewer doesn't understand the example \"a reviewer giving a strong accept to two papers...\". Could you explain more?\n\n2) Page 7 Paragraph 1: The discussion around principle 3 is too brief. Some examples would help the readers.\n\n3) Page 11, Paragraph 2: \".. values of lambda in [Q]uantization validation\" should the Q be in lower case?\n\n4) Page 11, Section 5.2, 2: \"We make simple adjustments to the baseline..\" it's better to briefly summarize the nature of the adjustments to save the readers a trip to the appendix.\n\n5) Page 12, Paragraph 4: Why do we want to look at the L2 error? It's better to give motivation.\n\n6) Page 13 Section 6: It's hard for The Reviewer to understand this sentence: \"it should not inadvertently reveal the review information of paper(s) outside their scope\". \n\n", "broader_impact_concerns": "The Reviewer doesn't have any concern about the potential negative impact of the paper. It's limited to the problem of the research community, and the methodology doesn't show any ethical or societal concerns.\n\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zS1KqZGqOn", "reviewer": "Reviewer_Mh5n", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Stregnths\n1. The studied problem has value in the peer review process.\n2. The formulation is sound under intuitive principles.\n3. The discussion in terms of connection to existing methods and analysis of special cases contribute to the understanding of the framework.\n\nWeakness\n1. The major concern is in terms of experiments. TMLR explicitly asks reviewers to evaluate papers via\"Are the claims made in the submission supported by accurate, convincing and clear evidence?\". The reviewer feels the paper does not fully meet the criterion on several aspects:\na. The paper does not really experiment on real-world datasets. \nb. Besides sythetic dataset, only ICLR2017 dataset is used. \nc. Only one non-trivial baseline is compared with. \n2. One very important hyperparameter in this work is lambda as in Algorithm 1. The selection process of lambda is somehow concerning.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the practical problem of generating de-quantized scores in the peer review process, so that area chairs can make more informed decisions. The paper proposes an algorithmic approach to integrate (partial) ranking, provided by each reviewer, into the de-quantized score generation process. The proposed algorithm is a constrained convex optimization problem that can be solved by existing solvers, so the main contribution of this work is the formulation (not an algorithm), which follows three design principals around consistency and consensus. The paper describes its connection to existing models as special cases (e.g., when only scores are available). Experiments are conducted in two synthetic settings: 1) synthetic dataset where data is generated under specific statistical models. 2) \"real-world\" ICLR2017 dataset. The proposed method can outperform a naive baseline (observed score without any modification, and an adjusted version of an existing method).", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "TMLR explicitly asks reviewers to evaluate papers via\"Are the claims made in the submission supported by accurate, convincing and clear evidence?\". The reviewer feels the paper needs more work to fully meet the criterion:\n\n1. The paper does not really experiment on real-world datasets. The ICLR dataset was artificial in several aspects 1) The scores were used as groundtruth, and another level of quantization was used to generate \"observed scores\". 2) Reviewer assignments were random. The reviewer understands the difficulty of getting a more ideal dataset, but as a scientific work those difficulties should probably be mitigated nevertheless. Or the authors should elaborate on why and how such artificial settings transfer to the real-world. \n\n2. Besides sythetic dataset, only ICLR2017 dataset is used. There are several datasets in the Kang et al. paper. Please validate the mehods on more datasets or elaborate on why using such datasets is hard. One difficulty the reviewer can see is, only ICLR provides level-10 rating, which allows further artificial quantization and feasible tuning of the hyperparameter (see next point). In summary, only one artificial dataset may not suffice as convincing evidence. The authors are suggested to provide more convincing results.\n\n3. Please compare with more baselines or elaborate on why other baselines are not feasible. \n\n4. One very important hyperparameter in this work is lambda as in Algorithm 1. Since there is no learning in this paper (i.e., existing solver will be applied directly, lambda is pretty much the only knot that controls the solution). This hyperparameter can not be tuned on validation dataset which does not exist, so this paper proposes a QV method, which basically does another level of quantization. There are a couple of concerns that the authors should provide more evidence on why the solution is feasible and practical: a) the other level of quantization clearly requires the original score range not to be small. For example, many conferences only have 5 levels of scores (most scores will be around borderline) and another level  of quantization will likely to lose too much information to make the selection valid. Please elaborate on this point. b). The further quantization is under specific quantization assumptions - please elaborate on why and when the assumptions can transfer to real-world settings. ", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wL7YUFmYlP", "reviewer": "Reviewer_J9y1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n------------\n(1) Clear writing\n\n(2) Good motivation\n\n(3) Realistic setting\n\n(4) Experiments are a good start\n\nWeaknesses\n----------------\n(1) It would be good to have one more stronger baseline. The two you have are essentially \"no algorithm at all\" and \"we set $\\lambda$ to infinity even though our $\\lambda$-selection procedure wouldn't have chosen it\". Although it is not a given that these two would be worse than your proposed algorithm, it is not surprising. What about something extremely simple such as:\n\nIf a reviewer gives both paper $A$ and $B$ the same score $r$, but ranks $A$ > $B$, then replace the scores assigned by $A$ and $B$ with $r + c \\cdot \\verb|np.random.randn()|$ and $r - c \\cdot \\verb|np.random.randn()|$, respectively. The constant $c$ could be cross-validated in the same you're doing now with $\\lambda$. The random number maybe should be the same one and not generated twice. This algorithm will (a) break ties and (b) take into account rankings, (c) is really easy to implement, but it's sloppy and has no theory behind it, so (hopefully) your algorithm will be superior.\n\n(2) The first simulation has papers assigned at random to reviewers. In reality, there is some affinity between papers and reviewers (\"expertise\") which is taken into account when papers are assigned. This could be expressed as different levels of noise $\\sigma$, where a \"good\" paper-reviewer match is low $\\sigma$ and a \"bad\" paper-reviewer match is a high $\\sigma$ and perhaps even a biased score where $y_{rp}$ is drawn from a Gaussian with mean $x^*_p + b$ where $b$ is the bias. \n\n\n\nQuestions\n------------\nFigure 2(a): what happens if you extend the x-axis to $\\sigma = 2$? Does BRE-adjusted eventually do better than proposed? If so, why?\n\n(2) Section 5.4, \"Since the reviewers are all anonymous in this dataset, we generate the reviewer assignments in a random manner.\" Is this for when you vary the number of papers per reviewer? Please clarify in the text.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Thanks to the authors for the hard work on this paper.\n\nThe authors address the problem of aggregating quantized scores when ranking are available for the purpose of having higher-quality peer review scores, and (as a result) fewer ties. They use a convex optimization procedure to combine quantized scores and partial rankings into estimated dequantized scores. Simulated and semi-simulated experiments are conducted showing that their algorithm works against two baselines. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "(1) This is critical for acceptance. Please implement the \"fool's\" baseline that I described above. I haven't thought about this for nearly as much time as you have, so there are likely to be obvious reasons why the above baseline is clearly bad. If so, I request that either (a) you justify why this baseline should not even be tried or (b) fix it and add its results to the paper. If you have an idea for a simple-but-reasonable baseline that you think makes more sense, please implement that one instead.\n\n(2) Try \"expertise-varying\" simulations as described in the preceding section. Not required for acceptance, but recommended.\n\n(3) Add a new line to (some of) the error plots that shows the proposed algorithm under the oracle $\\lambda$. This would strengthen the conclusions of Figure 4 by further showing that your $\\lambda$ selection scheme only pays a small penalty. Not required for acceptance, but recommended.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an algorithm that can help area chairs to make better decisions. The algorithm combines a quantized score of the paper with a partial ordering of the papers into a single de-quantized score that can be acted upon. The problem of computing the de-quantized score is formulated as a convex optimization problem. The proposed method is evaluated on synthetic problems and also on ICLR 2017 reviews.

This paper was well received and is also well written. There were two kinds of major comments:
1. Insufficient baselines and experimental evaluations. The authors addressed these comments by running more experiments and including them in the paper.
2. All experiments are conducted only on one real-world dataset of ICLR 2017 reviews. The authors clearly explained that all other paper review datasets do not have the properties that they seek. The datasets either do not have review scores or the granularity of the scores is insufficient to evaluate the proposed method. Given this, my opinion is that the empirical evaluation is sufficient.

The only thing that I ask the authors for is that they include a paragraph that describes the limitations of the other datasets in the paper. I believe that the readers of the paper will find it interesting. Since this is easy to do, simply by copying the paragraph from the rebuttals, this paper is "accepted as is". Congratulations!
Decision: Accept (poster)