You Are the Best Reviewer of Your Own Papers: An Owner-Assisted Scoring Mechanism

1 Introduction . Designing mechanisms for evaluating a set of items by aggregating scores from agents has applications in a wide range of domains [ 7 , 2 , 25 ] . As an example of these applications , reviewers are asked to score submissions for conference proceedings on the basis of certain criteria , and the scores serve as a crucial factor in accepting or rejecting the submissions . Ideally , one wishes to design a mechanism that is expected to produce scores that adequately reflect the quality of the submissions . In the last few years , however , the explosion in the number of submissions to many artificial intelligence and machine learning conferences has made it very challenging to maintain high-quality scores from peer review . For instance , the number of submissions to NeurIPS—a leading machine learning conference—has grown from 1,673 in 2014 to a record-high of 9,454 in 2020 [ 18 , 13 ] . On the other hand , the number of qualified reviewers—those , for example , who have published at least one paper in a premier machine learning conference—simply can not keep up with the growth of the submission size [ 24 ] . This significant disparity imposes significant burdens on the qualified reviewers , thereby reducing the average amount of time spent on reviewing a paper and , worse , pushing program chairs to solicit novice reviewers for top-tier machine learning conferences , including NeurIPS , ICML , AAAI , and many others [ 26 ] . Given the fundamental role of peer review in scientific research , the consequence of this reality has started to impede the advance of machine learning research [ 17 , 8 , 28 ] . In the famous NIPS experiment in 2014 [ 18 ] , review scores are observed to have a surprisingly high level of arbitrariness . The situation has presumably become even worse , as the number of submissions in 2020 was 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . more than five times that when the experiment was conducted . While a moderate level of review arbitrariness reflects the diversity of the scientific community , such a high level as in the 2014 NIPS experiment would very likely lead to acceptance of many low-quality papers and , correspondingly , rejection of a high volume of high-potential papers . In the long term , the poor reliability of peer review might erode the public confidence in these machine learning conference proceedings , a challenge that the scientific community can not afford to ignore [ 23 ] . To address this challenge , in this paper I introduce a mechanism that allows one to produce more reliable scores that are modified from the raw review scores . While it seems unrealistic to extend the reviewer pool or assign more papers to each reviewer , it is possible—though rarely done—to extract information from the authors and incorporate it into the improvement of scores ( see , e.g. , [ 25 ] ) . In fact , an author often submits multiple papers simultaneously to the same conference and , moreover , is likely to be among the most capable persons to assess the quality of her submissions . Unfortunately , if asked , the author has an incentive to report higher scores than what she believes so as to maximize the utility . To reconcile this dilemma , instead , my mechanism requires the author to report a ranking of her submissions in decreasing order of the quality . With this ranking in place , the mechanism solves its adjusted scores from a simple convex programming problem—which is in the form of isotonic regression [ 4 ] —using the raw review scores and the ranking as data . Thus , this new mechanism is referred to as the Isotonic Mechanism . The Isotonic Mechanism comes with several appealing properties that guarantee its superiority over the use of the raw review scores for decision making . First , I show that a rational author ’ s unique optimal strategy is to report the true ranking of her submissions under the assumption that the utility of the author is an additive convex function over all the submissions . Even in the case where the author is not completely certain about the ranking of the “ true underlying ” scores , the author is better off reporting the most accurate ranking as far as she knows . Second , I prove that the Isotonic Mechanism supplied with the optimal strategy taken by the author strictly improves the accuracy of estimating the true underlying scores . Our further analysis reveals that the gain of the Isotonic Mechanism over the raw scores becomes especially substantial if the variability of the scores provided by reviewers is high and the number of submissions by the author is large—the very situation that many machine learning conferences are faced with . While the high variability often results from the reduction of review time per submission and the arbitrariness of novice reviewers [ 10 ] , more recently it has become increasingly common for an author to submit multiple papers to one single conference . For example , there are 133 authors who submitted at least five papers to ICLR in 2020 , which constitutes a significant portion of the total 2,594 submissions . Even more surprisingly , the maximum number of submissions authored by one single individual is as high as 32 in this conference [ 27 ] . These ever-growing statistics render the Isotonic Mechanism an ideal approach to improving peer review in these scenarios . In relating to the literature , the Isotonic Mechanism is largely orthogonal to many existing approaches to obtaining better review scores [ 19 , 11 , 29 ] . In this line of research , the primary focus is to develop mechanisms that incentivize reviewers to provide more accurate scores or to sharpen the scores by modeling the generation of scores . In stark contrast , the Isotonic Mechanism solely exploits the information from the authors ’ side in addition to the raw scores . This orthogonality suggests the promise of incorporating the underlying idea of the Isotonic Mechanism into many existing approaches . For completeness , I remark that an emerging line of research in the mechanism design area leverages additional author information to improve peer review [ 1 , 22 , 14 , 21 ] . 2 Methodology . In this section , I introduce the Isotonic Mechanism in detail , and provide theoretical guarantees under certain conditions showing that ( 1 ) the owner/author would report her true ranking of her items/submissions in order to maximize the expected utility , and ( 2 ) this mechanism improves the overall statistical estimation efficiency for evaluating the papers . Throughout this paper , I consider an owner of the items who is fully rational in that her ultimate goal is to maximize the expectation of the overall utility as much as she can . Consider n items and let their true scores be denoted by R1 , R2 , . . . , Rn . The reviewers rate the n items as y1 , . . . , yn , which are given by yi = Ri + zi ( 1 ) for i = 1 , . . . , n. Above , zi ’ s are the noise terms . In our setup , the owner of the n items is asked to provide a ranking of the items , denoted by π , which is a permutation of 1 , 2 , . . . , n. Ideally , the permuted true scores π ◦R : = ( Rπ ( 1 ) , Rπ ( 2 ) , . . . , Rπ ( n ) ) are ( approximately ) in the nonincreasing order , but it does not necessarily have to be this case , as the owner has complete freedom to report any ranking . In our mechanism , the ranking serves as a shaperestricted constraint that shall facilitate better estimation of the true underlying scores R1 , . . . , Rn based on the raw scores y = ( y1 , . . . , yn ) . More precisely , taking as input the raw scores y by the reviewers and the ranking π by the owner , our mechanism outputs scores R̂ : = ( R̂1 , . . . , R̂n ) that either serve as the adjusted scores or as a reference for the decision-making by , for example , the program chair . Now I introduce the Isotonic Mechanism for adjusting the raw scores by the reviewers , provided the input data y and ranking π . Throughout the paper , ‖ · ‖ denotes the Euclidean norm . The Isotonic Mechanism reports R̂ ( π ) , which is the optimal solution of the following convex program : min r 1 2 ‖y − r‖2 s.t . rπ ( 1 ) ≥ rπ ( 2 ) ≥ · · · ≥ rπ ( n ) . ( 2 ) When clear from the context , I omit the dependence on π when writing the reported ranking R̂ . In words , the Isotonic Mechanism finds the solution by projecting the raw scores onto the feasible region { r : rπ ( 1 ) ≥ rπ ( 2 ) ≥ · · · ≥ rπ ( n ) } , which constitutes a convex cone that is often referred to as the isotonic cone ( up to permutation ) , hence the name Isotonic Mechanism . Needless to say , the mechanism wishes that the owner can report the true ranking , although it is not necessarily the case . If the reported ranking is π ? , the true values R is a feasible point of the program ( 2 ) and this program is essentially a restricted least-squares estimator . The projection shall presumably improve the estimation accuracy as it greatly reduces the search space for the true scores . From an algorithmic angle , an appealing feature of this mechanism is that ( 2 ) is a convex quadratic programming problem and , therefore , is tractable . Even better , this program admits a certain structure that allows for a very efficient algorithm called the pool adjacent violators algorithm [ 16 , 3 , 6 ] . Interested readers are referred to [ 5 , 12 ] for other algorithms for solving isotonic regression . To proceed , several pressing questions concerning the Isotonic Mechanism must be addressed . What is the optimal strategy of the owner and will she report the true ranking out of her own interest ? Moreover , does the mechanism , provided that the owner chooses the optimal strategy , really outperform the raw scores of the reviewers in any sense ? These questions will be addressed by the following subsections . 2.1 Optimality . To find the optimal strategy of the owner , I first recognize that the owner is rational and aims to maximize her expected utility by supplying any ranking π of her choice to the Isotonic Mechanism ( 2 ) . On the surface , it seems unclear which ranking would maximize the expected utility . I make the following assumptions for the setup for the mechanism . Before introducing the assumptions , it is worth noting that the next section extends our results to a certain relaxed version of these assumptions . Assumption 2.1 . Given ( final ) scores R̂1 , . . . , R̂n of n items in the possession of the owner , then the owner ’ s utility takes the form Util ( R̂ ) = ∑n i=1 U ( R̂i ) , where U is a nondecreasing convex function . In this assumption , the utility function is separable in the sense that it is additive over all the n items . Regarding the convexity assumption of the utility function , note that it boils down to requiring that the marginal utility U ′ ( r ) ( assume differentiability ) is a nondecreasing function . At first glance , it is not consistent with the conventional law of diminishing marginal utility [ 15 ] . However , a more careful look first shows that score R̂i does not measure the quantity of some matter . Moreover , note that for peer review in conference proceedings , a high score largely determines whether the paper would be presented as a poster , oral talk , or even as a plenary talk , which really makes a significant difference for a paper in terms of impact . In this spirit , the utility increases rapidly as the score increases and therefore a convexity assumption on the utility sheds light on this reality . In Section 3 I consider an extension of this assumption . In addition , I discuss this assumption in the practical context of peer review in Section 4 . Notably , the utility in many applications can vanish to zero once the score r is below some value . In view of this fact , examples of such utility functions include U ( r ) = max { 0 , ar + b } and U ( r ) = max { 0 , ear+b − c } , where a and c are positive constants . Assumption 2.2 . The owner has knowledge of the true ranking of her items . That is , the owner knows which permutation π ? that makes π ? ◦R in nonincreasing order . Remark 2.1 . Turning to Assumption 2.2 , it is worthwhile mentioning that the owner only needs to compare the true values instead of knowing the exact values of Ri ’ s in order to report the true ranking . In Section 3 , I relax this assumption by considering an owner who possesses only partial knowledge of the true ranking . Assumption 2.3 . The noise ( z1 , . . . , zn ) follows an exchangeable distribution in the sense that ( z1 , . . . , zn ) has the same probability distribution in Rn as ρ ◦ z : = ( zρ ( 1 ) , . . . , zρ ( n ) ) for any permutation ρ of 1 , . . . , n. Remark 2.2 . This assumption includes the often-assumed setting where ( z1 , . . . , zn ) is comprised of independent and identically distributed random variables . This generality is useful in the case where the noise terms involve a common latent factor—for example , a recent trend in research directions—which invalidates independence but the exchangeability remains true . Furthermore , it is beneficial to note that if the score of the i-th paper is averaged over several reviewers , the noise terms may have different variances . However , our assumption remains valid if I can treat the number of reviewers assigned to each paper as an independent and identically distributed random variable . In addition , technically speaking , our assumptions do not require the noise to have zero mean , though I do not intend to emphasize this generalization . Interestingly , the following theorem shows that reporting the true ranking is just the optimal strategy . Recall that the true ranking π ? is a permutation of 1 , . . . , n such that the permuted true scores obey Rπ ? ( 1 ) ≥ Rπ ? ( 2 ) ≥ · · · ≥ Rπ ? ( n ) , and the expected utility EUtil ( R̂ ) = E [ n∑ i=1 U ( β̂i ) ] ( 3 ) is taken over the randomness of the noise terms in ( 1 ) . Theorem 1 . Under Assumptions 2.1-2.3 , the expected utility ( 3 ) is maximized when the Isotonic Mechanism is provided with the true ranking π ? . That is , the owner ’ s optimal strategy is to honestly report the true ranking . To better appreciate this theorem , note that this optimal strategy is “ computationally free ” in the sense that reporting the true ranking does not require further effort from the owner under our assumptions . In particular , the optimal strategy does not rely on the raw scores provided by the reviewers . Moreover , as revealed by the proof in Section 2.2 , if the true scores R do not contain ties and the function U is strictly convex , the optimality claimed in Theorem 1 is strict in the sense that EUtil ( R̂ ( π ) ) < EUtil ( R̂ ( π ? ) ) for any ranking π that is not identical to π ? . The gap between the two utility terms can be arbitrarily large when the true underlying scores are widely different from each other . Even in the presence of ties , the inequality above remains strict unless π is also a true ranking , thereby only differing from π ? in the indices corresponding to the ties . In light of Theorem 1 , in the remainder of this section I assume that the owner reports the true ranking out of self-interest . The next step is to study the performance of the Isotonic Mechanism from a social welfare viewpoint . Explicitly , for example , are the adjusted scores more accurate than the raw scores ? The following theorem answers this question in the affirmative . Theorem 2 . Under Assumptions 2.1-2.3 , the Isotonic Mechanism improves the estimation accuracy of the true underlying scores in the sense that E [ n∑ i=1 ( R̂i ( π ? ) −Ri ) 2 ] ≤ E [ n∑ i=1 ( yi −Ri ) 2 ] . Remark 2.3 . Above , the ` 2 loss is chosen mainly for technical convenience . An interesting question for future research is to analyze the performance using other loss functions . Together with Theorem 1 , Theorem 2 demonstrates the superiority of the Isotonic Mechanism over the use of the raw scores by the reviewers . Furthermore , the following result shows that the gained improvement becomes more significant in the case when the number of items and the noise variance are large . To state this theorem , below I impose more refined assumptions on the noise terms . Besides , I let V ( R ) : = n−1∑ i=1 |Rπ ? ( i+1 ) −Rπ ? ( i ) | = max i Ri −min i Ri denote the total variation of R. Theorem 3 . In additions to Assumptions 2.2 and 2.1 , assume that z1 , . . . , zn are independent and identically distributed normal random variables N ( 0 , σ2 ) . Letting both σ > 0 and V > 0 be fixed , I have 0.4096 + on ( 1 ) ≤ supR : V ( R ) ≤V E [ ∑n i=1 ( R̂i ( π ? ) −Ri ) 2 ] n 1 3σ 4 3V 2 3 ≤ 7.5625 + on ( 1 ) , where both on ( 1 ) → 0 as n→∞ . Remark 2.4 . This theorem is adapted from a well-known result of isotonic regression ( see , e.g. , Theorem 2.3 in [ 30 ] ) and , thus the proof is omitted . As an aside , the risk bounds should not be interpreted as being always increasing with the total variation bound V . To see this , note that when Rπ ? ( i+1 ) Rπ ? ( i ) for all i , then the raw scores y1 , . . . , yn satisfy the isotonic constraint ( 2 ) with π set to π ? with high probability . As a consequence , the risk of the Isotonic Mechanism is roughly equal to that of the raw scores . Even if the total variation V tends to infinity , the risk stays constant . Loosely speaking , this theorem says that the risk of the Isotonic Mechanism is of order O ( n1/3σ4/3 ) . For comparison , note that the risk of the raw scores is E [ ∑n i=1 ( yi −Ri ) 2 ] = E [ ∑n i=1 z 2 i ] = nσ2 . Recognizing their dependence on the noise level σ , I see that the Isotonic Mechanism is especially preferable when σ is large or , put differently , when the reviewers give very noisy scores . Moreover , the Isotonic Mechanism has a risk that grows much slower with respect to the number of items n .

Summary: This paper looks at the question improving the process of peer review by allowing authors to submit a ranking of their own papers. The idea is to use this ranking from the authors to improve the noisy rankings coming from the reviewers. The authors show that one can formulate this problem as an isotonic regression problem and that if the authors are utility maxamizers for the sum utility of a set of papers then it is rational to submit and hones ranking of the papers. Extensions are also proposed including authors submitting bucket rankings as well as variants of utility functions.