You Are the Best Reviewer of Your Own Papers: An Owner-Assisted Scoring Mechanism
Year: 2022

Abstract:
I consider the setting where reviewers offer very noisy scores for a number of items for the selection of high-quality ones (e.g., peer review of large conference proceedings) whereas the owner of these items knows the true underlying scores but prefers not to provide this information. To address this withholding of information, in this paper, I introduce the Isotonic Mechanism, a simple and efficient approach to improving on the imprecise raw scores by leveraging certain information that the owner is incentivized to provide. This mechanism takes as input the ranking of the items from best to worst provided by the owner, in addition to the raw scores provided by the reviewers. It reports adjusted scores for the items by solving a convex optimization problem. Under certain conditions, I show that the owner’s optimal strategy is to honestly report the true ranking of the items to her best knowledge in order to maximize the expected utility. Moreover, I prove that the adjusted scores provided by this owner-assisted mechanism are indeed significantly more accurate than the raw scores provided by the reviewers. This paper concludes with several extensions of the Isotonic Mechanism and some refinements of the mechanism for practical considerations.

heading: 1 Introduction
text: Designing mechanisms for evaluating a set of items by aggregating scores from agents has applications in a wide range of domains [7, 2, 25]. As an example of these applications, reviewers are asked to score submissions for conference proceedings on the basis of certain criteria, and the scores serve as a crucial factor in accepting or rejecting the submissions. Ideally, one wishes to design a mechanism that is expected to produce scores that adequately reflect the quality of the submissions.
In the last few years, however, the explosion in the number of submissions to many artificial intelligence and machine learning conferences has made it very challenging to maintain high-quality scores from peer review. For instance, the number of submissions to NeurIPS—a leading machine learning conference—has grown from 1,673 in 2014 to a record-high of 9,454 in 2020 [18, 13]. On the other hand, the number of qualified reviewers—those, for example, who have published at least one paper in a premier machine learning conference—simply cannot keep up with the growth of the submission size [24]. This significant disparity imposes significant burdens on the qualified reviewers, thereby reducing the average amount of time spent on reviewing a paper and, worse, pushing program chairs to solicit novice reviewers for top-tier machine learning conferences, including NeurIPS, ICML, AAAI, and many others [26].
Given the fundamental role of peer review in scientific research, the consequence of this reality has started to impede the advance of machine learning research [17, 8, 28]. In the famous NIPS experiment in 2014 [18], review scores are observed to have a surprisingly high level of arbitrariness. The situation has presumably become even worse, as the number of submissions in 2020 was
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
more than five times that when the experiment was conducted. While a moderate level of review arbitrariness reflects the diversity of the scientific community, such a high level as in the 2014 NIPS experiment would very likely lead to acceptance of many low-quality papers and, correspondingly, rejection of a high volume of high-potential papers. In the long term, the poor reliability of peer review might erode the public confidence in these machine learning conference proceedings, a challenge that the scientific community cannot afford to ignore [23].
To address this challenge, in this paper I introduce a mechanism that allows one to produce more reliable scores that are modified from the raw review scores. While it seems unrealistic to extend the reviewer pool or assign more papers to each reviewer, it is possible—though rarely done—to extract information from the authors and incorporate it into the improvement of scores (see, e.g., [25]). In fact, an author often submits multiple papers simultaneously to the same conference and, moreover, is likely to be among the most capable persons to assess the quality of her submissions. Unfortunately, if asked, the author has an incentive to report higher scores than what she believes so as to maximize the utility. To reconcile this dilemma, instead, my mechanism requires the author to report a ranking of her submissions in decreasing order of the quality. With this ranking in place, the mechanism solves its adjusted scores from a simple convex programming problem—which is in the form of isotonic regression [4]—using the raw review scores and the ranking as data. Thus, this new mechanism is referred to as the Isotonic Mechanism.
The Isotonic Mechanism comes with several appealing properties that guarantee its superiority over the use of the raw review scores for decision making. First, I show that a rational author’s unique optimal strategy is to report the true ranking of her submissions under the assumption that the utility of the author is an additive convex function over all the submissions. Even in the case where the author is not completely certain about the ranking of the “true underlying” scores, the author is better off reporting the most accurate ranking as far as she knows. Second, I prove that the Isotonic Mechanism supplied with the optimal strategy taken by the author strictly improves the accuracy of estimating the true underlying scores.
Our further analysis reveals that the gain of the Isotonic Mechanism over the raw scores becomes especially substantial if the variability of the scores provided by reviewers is high and the number of submissions by the author is large—the very situation that many machine learning conferences are faced with. While the high variability often results from the reduction of review time per submission and the arbitrariness of novice reviewers [10], more recently it has become increasingly common for an author to submit multiple papers to one single conference. For example, there are 133 authors who submitted at least five papers to ICLR in 2020, which constitutes a significant portion of the total 2,594 submissions. Even more surprisingly, the maximum number of submissions authored by one single individual is as high as 32 in this conference [27]. These ever-growing statistics render the Isotonic Mechanism an ideal approach to improving peer review in these scenarios.
In relating to the literature, the Isotonic Mechanism is largely orthogonal to many existing approaches to obtaining better review scores [19, 11, 29]. In this line of research, the primary focus is to develop mechanisms that incentivize reviewers to provide more accurate scores or to sharpen the scores by modeling the generation of scores. In stark contrast, the Isotonic Mechanism solely exploits the information from the authors’ side in addition to the raw scores. This orthogonality suggests the promise of incorporating the underlying idea of the Isotonic Mechanism into many existing approaches. For completeness, I remark that an emerging line of research in the mechanism design area leverages additional author information to improve peer review [1, 22, 14, 21].

heading: 2 Methodology
text: In this section, I introduce the Isotonic Mechanism in detail, and provide theoretical guarantees under certain conditions showing that (1) the owner/author would report her true ranking of her items/submissions in order to maximize the expected utility, and (2) this mechanism improves the overall statistical estimation efficiency for evaluating the papers. Throughout this paper, I consider an owner of the items who is fully rational in that her ultimate goal is to maximize the expectation of the overall utility as much as she can.
Consider n items and let their true scores be denoted by R1, R2, . . . , Rn. The reviewers rate the n items as y1, . . . , yn, which are given by
yi = Ri + zi (1)
for i = 1, . . . , n. Above, zi’s are the noise terms. In our setup, the owner of the n items is asked to provide a ranking of the items, denoted by π, which is a permutation of 1, 2, . . . , n. Ideally, the permuted true scores π ◦R := (Rπ(1), Rπ(2), . . . , Rπ(n)) are (approximately) in the nonincreasing order, but it does not necessarily have to be this case, as the owner has complete freedom to report any ranking. In our mechanism, the ranking serves as a shaperestricted constraint that shall facilitate better estimation of the true underlying scores R1, . . . , Rn based on the raw scores y = (y1, . . . , yn). More precisely, taking as input the raw scores y by the reviewers and the ranking π by the owner, our mechanism outputs scores R̂ := (R̂1, . . . , R̂n) that either serve as the adjusted scores or as a reference for the decision-making by, for example, the program chair.
Now I introduce the Isotonic Mechanism for adjusting the raw scores by the reviewers, provided the input data y and ranking π. Throughout the paper, ‖ · ‖ denotes the Euclidean norm.
The Isotonic Mechanism reports R̂(π), which is the optimal solution of the following convex program:
min r
1 2 ‖y − r‖2
s.t. rπ(1) ≥ rπ(2) ≥ · · · ≥ rπ(n). (2)
When clear from the context, I omit the dependence on π when writing the reported ranking R̂.
In words, the Isotonic Mechanism finds the solution by projecting the raw scores onto the feasible region {r : rπ(1) ≥ rπ(2) ≥ · · · ≥ rπ(n)}, which constitutes a convex cone that is often referred to as the isotonic cone (up to permutation), hence the name Isotonic Mechanism. Needless to say, the mechanism wishes that the owner can report the true ranking, although it is not necessarily the case. If the reported ranking is π?, the true values R is a feasible point of the program (2) and this program is essentially a restricted least-squares estimator. The projection shall presumably improve the estimation accuracy as it greatly reduces the search space for the true scores.
From an algorithmic angle, an appealing feature of this mechanism is that (2) is a convex quadratic programming problem and, therefore, is tractable. Even better, this program admits a certain structure that allows for a very efficient algorithm called the pool adjacent violators algorithm [16, 3, 6]. Interested readers are referred to [5, 12] for other algorithms for solving isotonic regression.
To proceed, several pressing questions concerning the Isotonic Mechanism must be addressed. What is the optimal strategy of the owner and will she report the true ranking out of her own interest? Moreover, does the mechanism, provided that the owner chooses the optimal strategy, really outperform the raw scores of the reviewers in any sense? These questions will be addressed by the following subsections.

heading: 2.1 Optimality
text: To find the optimal strategy of the owner, I first recognize that the owner is rational and aims to maximize her expected utility by supplying any ranking π of her choice to the Isotonic Mechanism (2). On the surface, it seems unclear which ranking would maximize the expected utility.
I make the following assumptions for the setup for the mechanism. Before introducing the assumptions, it is worth noting that the next section extends our results to a certain relaxed version of these assumptions.
Assumption 2.1. Given (final) scores R̂1, . . . , R̂n of n items in the possession of the owner, then the owner’s utility takes the form Util(R̂) = ∑n i=1 U(R̂i), where U is a nondecreasing convex function.
In this assumption, the utility function is separable in the sense that it is additive over all the n items. Regarding the convexity assumption of the utility function, note that it boils down to requiring that the marginal utility U ′(r) (assume differentiability) is a nondecreasing function. At first glance, it is not consistent with the conventional law of diminishing marginal utility [15]. However, a more careful look first shows that score R̂i does not measure the quantity of some matter. Moreover, note
that for peer review in conference proceedings, a high score largely determines whether the paper would be presented as a poster, oral talk, or even as a plenary talk, which really makes a significant difference for a paper in terms of impact. In this spirit, the utility increases rapidly as the score increases and therefore a convexity assumption on the utility sheds light on this reality.
In Section 3 I consider an extension of this assumption. In addition, I discuss this assumption in the practical context of peer review in Section 4.
Notably, the utility in many applications can vanish to zero once the score r is below some value. In view of this fact, examples of such utility functions include U(r) = max{0, ar + b} and U(r) = max{0, ear+b − c}, where a and c are positive constants. Assumption 2.2. The owner has knowledge of the true ranking of her items. That is, the owner knows which permutation π? that makes π? ◦R in nonincreasing order. Remark 2.1. Turning to Assumption 2.2, it is worthwhile mentioning that the owner only needs to compare the true values instead of knowing the exact values of Ri’s in order to report the true ranking. In Section 3, I relax this assumption by considering an owner who possesses only partial knowledge of the true ranking. Assumption 2.3. The noise (z1, . . . , zn) follows an exchangeable distribution in the sense that (z1, . . . , zn) has the same probability distribution in Rn as ρ ◦ z := (zρ(1), . . . , zρ(n)) for any permutation ρ of 1, . . . , n. Remark 2.2. This assumption includes the often-assumed setting where (z1, . . . , zn) is comprised of independent and identically distributed random variables. This generality is useful in the case where the noise terms involve a common latent factor—for example, a recent trend in research directions—which invalidates independence but the exchangeability remains true. Furthermore, it is beneficial to note that if the score of the i-th paper is averaged over several reviewers, the noise terms may have different variances. However, our assumption remains valid if I can treat the number of reviewers assigned to each paper as an independent and identically distributed random variable. In addition, technically speaking, our assumptions do not require the noise to have zero mean, though I do not intend to emphasize this generalization.
Interestingly, the following theorem shows that reporting the true ranking is just the optimal strategy. Recall that the true ranking π? is a permutation of 1, . . . , n such that the permuted true scores obey Rπ?(1) ≥ Rπ?(2) ≥ · · · ≥ Rπ?(n), and the expected utility
EUtil(R̂) = E [ n∑ i=1 U(β̂i) ] (3)
is taken over the randomness of the noise terms in (1). Theorem 1. Under Assumptions 2.1-2.3, the expected utility (3) is maximized when the Isotonic Mechanism is provided with the true ranking π?. That is, the owner’s optimal strategy is to honestly report the true ranking.
To better appreciate this theorem, note that this optimal strategy is “computationally free” in the sense that reporting the true ranking does not require further effort from the owner under our assumptions. In particular, the optimal strategy does not rely on the raw scores provided by the reviewers. Moreover, as revealed by the proof in Section 2.2, if the true scores R do not contain ties and the function U is strictly convex, the optimality claimed in Theorem 1 is strict in the sense that EUtil(R̂(π)) < EUtil(R̂(π?)) for any ranking π that is not identical to π?. The gap between the two utility terms can be arbitrarily large when the true underlying scores are widely different from each other. Even in the presence of ties, the inequality above remains strict unless π is also a true ranking, thereby only differing from π? in the indices corresponding to the ties.
In light of Theorem 1, in the remainder of this section I assume that the owner reports the true ranking out of self-interest. The next step is to study the performance of the Isotonic Mechanism from a social welfare viewpoint. Explicitly, for example, are the adjusted scores more accurate than the raw scores? The following theorem answers this question in the affirmative. Theorem 2. Under Assumptions 2.1-2.3, the Isotonic Mechanism improves the estimation accuracy of the true underlying scores in the sense that
E [ n∑ i=1 ( R̂i(π ?)−Ri )2] ≤ E [ n∑ i=1 (yi −Ri)2 ] .
Remark 2.3. Above, the `2 loss is chosen mainly for technical convenience. An interesting question for future research is to analyze the performance using other loss functions.
Together with Theorem 1, Theorem 2 demonstrates the superiority of the Isotonic Mechanism over the use of the raw scores by the reviewers. Furthermore, the following result shows that the gained improvement becomes more significant in the case when the number of items and the noise variance are large.
To state this theorem, below I impose more refined assumptions on the noise terms. Besides, I let
V (R) := n−1∑ i=1 |Rπ?(i+1) −Rπ?(i)| = max i Ri −min i Ri
denote the total variation of R. Theorem 3. In additions to Assumptions 2.2 and 2.1, assume that z1, . . . , zn are independent and identically distributed normal random variables N (0, σ2). Letting both σ > 0 and V > 0 be fixed, I have
0.4096 + on(1) ≤ supR:V (R)≤V E
[∑n i=1 ( R̂i(π ?)−Ri )2]
n 1 3σ 4 3V 2 3
≤ 7.5625 + on(1),
where both on(1)→ 0 as n→∞. Remark 2.4. This theorem is adapted from a well-known result of isotonic regression (see, e.g., Theorem 2.3 in [30]) and, thus the proof is omitted. As an aside, the risk bounds should not be interpreted as being always increasing with the total variation bound V . To see this, note that when Rπ?(i+1) Rπ?(i) for all i, then the raw scores y1, . . . , yn satisfy the isotonic constraint (2) with π set to π? with high probability. As a consequence, the risk of the Isotonic Mechanism is roughly equal to that of the raw scores. Even if the total variation V tends to infinity, the risk stays constant.
Loosely speaking, this theorem says that the risk of the Isotonic Mechanism is of order O(n1/3σ4/3). For comparison, note that the risk of the raw scores is E [∑n i=1 (yi −Ri) 2 ] = E [∑n i=1 z 2 i ] = nσ2. Recognizing their dependence on the noise level σ, I see that the Isotonic Mechanism is especially preferable when σ is large or, put differently, when the reviewers give very noisy scores. Moreover, the Isotonic Mechanism has a risk that grows much slower with respect to the number of items n.

heading: 2.2 Sketch of Proof
text: This section is devoted to proving Theorem 1 and we defer the proof of Theorem 2 to the Appendix. We need the following two lemmas for the proof of Theorem 1. Lemma 2.4. Let x = (x1, . . . , xn) y = (y1, . . . , yn) in the sense that x1 ≥ y1, x1 + x2 ≥ y1 + y2, . . . , x1 + · · ·+ xn−1 ≥ y1 + · · ·+ yn−1 and x1 + · · ·+ xn = y1 + · · ·+ yn. Let x+ and y+ be the projections of x and y onto the isotonic cone {r : r1 ≥ r2 ≥ · · · ≥ rn}, respectively. Then, we have x+ y+. Lemma 2.5 (Hardy–Littlewood–Pólya inequality). Let f be a convex function. Assume that x and y are nonincreasing vectors (that is, x1 ≥ · · · ≥ xn and y1 ≥ · · · ≥ yn) and x y. Then, we have∑n i=1 f(xi) ≥ ∑n i=1 f(yi).
Lemma 2.5 is a well-known result in the theory of majorization. See its proof in [20].
In contrast, Lemma 2.4 is new to the literature and its proof requires some novel elements. This is partly because the definition of majorization in this lemma is slightly different from that in the literature [20]. In the literature, x y if the condition in Lemma 2.4 holds after ordering both x,y from the largest to the smallest. To this end, we need the following definition and Lemma 2.7. Definition 2.6. We call z1 an upward swap of z2 if there exists 1 ≤ i < j ≤ n such that z1k = z2k for all k 6= i, j and z1i + z1j = z2i + z2j , z1i ≥ z2i .
This definition amounts to saying that z1 is an upward swap of z2 if the former can be derived by transporting some mass from the entry of z2 to an earlier entry. It is easy to check that z1 z2 if z1 is an upward swap of z2.
Lemma 2.7. Let x y. Then there exists an integer m and z1, . . . ,zm such that z1 = x, zm = y, and zl is an upward swap of zl+1 for l = 1, . . . ,m− 1.
Due to space constraints, we relegate the proofs of Lemma 2.7 and Lemma 2.4 to the Appendix.
Proof of Theorem 1. Without loss of generality, assume that R1 ≥ R2 ≥ · · · ≥ Rn. In this case, the Isotonic Mechanism is
min 1
2 ‖y − r‖2
s.t. r1 ≥ r2 ≥ · · · ≥ rn,
where y = R+ z. As is clear, the solution, denoted R̂, is obtained by projecting y onto the isotonic cone C = {r : r1 ≥ r2 ≥ · · · ≥ rn}. Now consider that the owner proposes a different ranking, π, which corresponds to
min 1
2 ‖y − r‖2
s.t. rπ(1) ≥ rπ(2) ≥ · · · ≥ rπ(n), which is equivalent to
min 1
2 ‖π ◦ y − r‖2
s.t. r1 ≥ r2 ≥ · · · ≥ rn after an appropriate permutation operation. In this case, the owner’s ratings be can obtained by projecting yπ onto the isotonic cone, and then pulling back the original permutation by applying π−1. Now, defining π ◦ a as π ◦ a = (aπ(1), aπ(2), . . . , aπ(n)) for a vector a, we observe that π ◦ y = π ◦R+ π ◦ z. For simplicity, we let ProjC denote the projection operator. Consider the two projections associated with the two rankings, namely ProjC(y) and π
−1 ◦ ProjC(π ◦ y). Due to our construction of the utility function, the owner is indifferent between π−1 ◦ ProjC(π ◦ y) and ProjC(π ◦y). Next, making use of our assumption on the noise distribution, we have π ◦R+π ◦z d = π ◦ R + z. Now we compare ProjC(R + z) and ProjC(π ◦ R + z) in terms of majorization. Using the simple but important fact that R + z π ◦ R + z, it follows from Lemma 2.4 that ProjC(R+ z) ProjC(π ◦R+ z). Therefore, Lemma 2.5 gives
n∑ i=1 U (ProjC(R+ z)i) ≥ n∑ i=1 U (ProjC(π ◦R+ z)i) ,
from which we readily get
E [ n∑ i=1 U (ProjC(R+ z)i) ] ≥ E [ n∑ i=1 U (ProjC(π ◦R+ z)i) ] .
That is, we have Utiltrue ranking ≥ Utilπ for an arbitrary ranking π. This finishes the proof.

heading: 3 Extensions
text: In this section, we extend our main results by relaxing the assumptions made in Section 2.1. Our primary aim is to show that the rational owner would continue to report the true ranking of her items in more general settings. Omitted proofs are deferred to the Appendix.
Ranking in a block form. Suppose that the owner only knows partial information of the true ranking in a block sense: let n1, . . . , nm be positive integers satisfying n1 + · · ·+ nm = n and {1, 2, . . . , n} be partitioned into {1, 2, . . . , n} = I?1 ∪ I?2 ∪ · · · ∪ I?m such that |Ii| = ni for i = 1, . . . ,m. The owner knows the true ranking satisfies
RI?1 ≥ RI?2 ≥ · · · ≥ RI?m , (4)
but the ranking within each block is completely unknown to the owner. The owner is asked to report an (ordered) partition I1, . . . , Im of {1, . . . , n} such that |Ii| = ni for i = 1, . . . ,m. Now we consider the following (n1, . . . , nm)-block Isotonic Mechanism:
min r
1 2 ‖y − r‖2
s.t. rI1 ≥ rI2 ≥ · · · ≥ rIm . (5)
Note that it is also a convex optimization program. Assumption 3.1. The owner has knowledge of the true (n1, . . . , nm)-block ranking of her items. That is, the owner knows which (n1, . . . , nm)-sized partition that satisfies (4). Theorem 4. Under Assumptions 2.1, 3.1, and 2.3, the expected utility (3) is maximized when the block Isotonic Mechanism is provided with the true (n1, . . . , nm)-sized (ordered) partition. Remark 3.1. The theorem above can intuitively be extended in a hierarchical manner. That is, for each block Ii, the owner can further provide a ranking within this block either in a partial sense or complete sense. The theorem shall continue to be valid and a rigorous treatment of this extension is left for future research.
Denote by I := (I1, . . . , Im). Let πI,y be formed by padding the m blocks while setting the entries of y within each block in nonincreasing order. That is, the set {πI,y(1), . . . , πI,y(n1)} = I1 and yπI,y(1) ≥ yπI,y(2) ≥ · · · ≥ yπI,y(n1), and likewise for the following m − 1 blocks. For example, let n = 4, I1 = {1, 3}, I2 = {2, 4}, and y = (4.4, 6.6, 5,−1), then (πI,y(1), πI,y(2), πI,y(3), πI,y(4)) = (3, 1, 2, 4) and yπI,y = (5, 4.4, 6.6,−1).
Robustness to inconsistencies. Next, we extend Theorem 1 to the setting where the owner might give a ranking that is not consistent with the true values. Imagine that the owner is faced with the problem of choosing between two rankings π1, π2. While π1 nor π2 might not render the true values R in nonincreasing order, we assume that the former is more faithful with respect to R than the latter in the sense that π1 ◦R π2 ◦R. (6) To better understand when this condition holds, we say that π1 is an upward shuffle of π2 if there exists two indices 1 ≤ i < j ≤ n such that
Rπ1(i) = Rπ2(j) > Rπ1(j) = Rπ2(i)
and π1(k) = π2(k) for all k 6= i, j. As is clear, (π1, π2) satisfies (6). In general, (6) is satisfied if π1 can be derived by sequentially upwardly shuffling from π2, by recognizing the transitivity of majorization. However, it is worth noting that it is not correct that any pair (π1, π2) satisfying (6) can always be sequentially shuffled from one to another. An example is π1 ◦R = (100, 0, 1, 10) and π2 ◦R = (10, 1, 0, 100). The following result demonstrates that Theorem 1 extends to this fault-tolerant setting. Theorem 5. Under Assumptions 2.1 and 2.3, the owner in the face of choosing between π1 and π2 satisfying (6), which is known to the owner, would be better off reporting π1 rather than π2 in terms of the overall utility.
Non-separable utility functions. A function f : Rn → R is called a Schur-convex function if it is symmetric in its n coordinates and f(x) ≥ f(y) for any x,y such that x1 ≥ · · · ≥ xn, y1 ≥ · · · ≥ yn and x y as defined in Lemma 2.4. As shown by the proofs in Section 2.2, the utility Util(r) given in Assumption 2.1 is Schur-convex. More generally, when f is differentiable and symmetric, then f is Schur-convex if and only if
(ri − rj) ( ∂f(r)
∂ri − ∂f(r) ∂rj
) ≥ 0
for all r ≡ (r1, . . . , rn) (see, e.g., [9]). Recognizing that the final step in the proof of Theorem 1 relies on the majorization property, we readily obtain the following result. Proposition 3.2. In addition to Assumptions 2.2 and 2.3, assume that the function Util(r) is Schurconvex. Then, the owner’s optimal strategy is to honestly report the true ranking of her items.
True score-dependent utility. Intuitively, the utility of each items shall depend on the true score of the item. An owner may value more about the acceptance of a high-quality item. In this spirit, we consider utility function that takes the following form:
Util(R̂) = n∑ i=1 U(R̂i;Ri), (7)
where we assume that U(r;R) is convex is in its first argument and satisfies
∂U(r;R) ∂r ≥ ∂U(r;R
′)
∂r
whenever R > R′. For example, consider U(r;R) = g(r)h(R) where g ≥ 0 is convex and h ≥ 0 is a nondecreasing differential function.
The following result shows that Theorem 1 remains valid under this generalization. Proposition 3.3. In addition to Assumptions 2.2 and 2.3, assume that the function Util(r) is given in (7). Then, the owner’s optimal strategy is to honestly report the true ranking of her items.
Proof of Proposition 3.3. The proof of this proposition is basically the same as that of Theorem 1 except for a minor modification. Explicitly, observe that
n∑ i=1 U(R̂i;Ri)
is always upper bounded by n∑ i=1 U(R̂i;Rρ(i))
for a permutation ρ such that R̂i’s and Rρ(i)’s have the same ranking in decreasing order. An important fact is that in reporting the true ranking, the utility has already been maximized over all permutations. The remaining part of the proof is the same as that of Theorem 1.
Soft constraints. Conceivably, one would argue that the hard constraints provided by the isotonic constraint seems too strong, especially in the case where the owner might not be completely sure about the ranking of her items. In this case, a simple adjustment is to consider the following convex combination as a candidate of scores:
R̂′ := θR̂(π?) + (1− θ)y, where 0 < θ < 1. Owing to the convexity of the risk, Theorem 2 remains correct. Proposition 3.4. Under the assumptions of Theorem 2, we have
E [ n∑ i=1 ( R̂′i −Ri )2] ≤ E [ n∑ i=1 (yi −Ri)2 ] .
Note that this generalization does not change the mechanism in that the owner still reports the ranking in view of the isotonic regression. More generally, we consider the following optimization program:
min r
1 2 ‖y − r‖2 + Pen(r),
where the (nonnegative) penalty term Pen(r) encourages isotonic monotonicity in the sense that, for example, Pen(r) = 0 if r is in the isotonic cone C and otherwise Pen(r) > 0. A choice of this penalty is
Pen(r) = λ n−1∑ i=1 (rπ(i+1) − rπ(i))+,
where λ > 0, π is the ranking reported by the owner and x+ := max{x, 0}. As λ → ∞, the penalty of the violation of the isotonic constraint tends to infinity. Therefore, we have the following simple fact.
Proposition 3.5. The Isotonic Mechanism can be (asymptotically) recovered by setting λ→∞ in (8).
However, the coupling strategy is invalid when it comes to proving for this new mechanism. While can we prove that Theorem 1 holds for this λ-dependent mechanism when n = 2, we leave the proof or disproof of the general case n ≥ 3 for future research. Proposition 3.6. When n = 2, the mechanism above is truth-optimal.

heading: 4 Discussion
text: In this paper, I have introduced the Isotonic Mechanism to incentivize the author of a set of papers to report the true ranking, thereby obtaining more accurate scores for decision making. The mechanism is conceptually simple and computationally efficient. Owing to the appealing optimality guarantees of this mechanism, it is likely that the use of this mechanism can to some extent alleviate the poor quality of review scores in many large machine learning conferences. Toward the employment of this mechanism, an intermediate step is to test the use of this mechanism. For example, conference organizers can require that a randomly selected group of authors submit the ranking and use the outcome of the Isotonic Mechanism for this selected group as a test.
In closing, I propose several directions to tackle practical challenges in applying the Isotonic Mechanism. First, the assumption of convex utility, which is critical for the optimality of the mechanism, might not always hold for some authors. As argued in Section 2.1, the utility is convex or approximately convex if the author aims for higher scores in order to be considered for more visibility and paper awards, beyond the acceptance of her papers. In contrast, if nothing more than acceptance or rejection makes a difference to the author, one may turn to a thresholded utility function of the form
U(r) = { 0, if r < r0, u, if r ≥ r0,
where u and r0 are some constants. An important question for future work is, therefore, to extend the Isotonic Mechanism in the face of non-convex utility.
From a practical standpoint, another direction is to incorporate strategic behaviors of both sides of the author and reviewers. The author might deliberately submit additional very low-quality papers and rank these as the worst. In this case, the Isotonic Mechanism tends to push upward the scores for “normal” papers, thereby increasing the chance of acceptance for these papers which the author cares about. A possible approach to overcoming this issue is to set a screening step to filter very low-quality submissions. Moreover, the Isotonic Mechanism enables the reviewers to have an influence on the author’s other papers that are examined by a different set of reviewers. It is interesting to investigate how the reviewers would act for self-interest and improve on the mechanism to confront any violation of ethical codes.
Third, while Theorem 3 shows the great benefits of using the Isotonic Mechanism, it would be of interest to perform experiments to quantify the magnitude of the benefits. As of finishing this paper, the NeurIPS 2021 Program Chairs asked each author to “rank their papers in terms of their own perception of the papers’ scientific contributions to the NeurIPS community,” though the data was not used for making decisions. Nevertheless, an interesting empirical study is to investigate what would be the outcome if the Isotonic Mechanism were used for NeurIPS 2021.
Last, a critical advance is to extend the Isotonic Mechanism to the setting where each item/paper is owned/authored by multiple owners/authors. This is a necessary step toward applying this mechanism to large machine learning conferences, where most papers are written by multiple authors. Simply invoking the Isotonic Mechanism for each author does not seem to be a good solution. For example, an author can adversarially rank a good paper co-authored by, say Bob, as the lowest so as to put her papers that are not co-authored by Bob in an advantageous position. From a different angle, would the use of the Isotonic Mechanism in an appropriate form discourage guest authorship? I leave this challenging extension for future work.
Limitations and societal impacts. A clear limitation of our work lies in its assumptions which our theoretical results rely on. Given the pivotal role of peer review in conference proceedings and the growing impact of AI on our society, we believe that the Isotonic Mechanism would have substantial societal impacts, provided that it would be used someday.
Acknowledgments and Disclosure of Funding
I am grateful to Nihar Shah and Haifeng Xu for very helpful and enlightening discussions. I would also like to thank the anonymous reviewers for their constructive comments that helped improve the presentation of this work. This work was supported in part by NSF through CAREER DMS-1847415 and CCF-1934876, an Alfred Sloan Research Fellowship, and the Wharton Dean’s Research Fund.

Review summaries:
This paper looks at the question improving the process of peer review by allowing authors to submit a ranking of their own papers. The idea is to use this ranking from the authors to improve the noisy rankings coming from the reviewers. The authors show that one can formulate this problem as an isotonic regression problem and that if the authors are utility maxamizers for the sum utility of a set of papers then it is rational to submit and hones ranking of the papers. Extensions are also proposed including authors submitting bucket rankings as well as variants of utility functions.

The author studies the problem of accurate evaluation of papers via peer review. They propose a mechanism by which authors may submit rankings of their submissions ordered by quality, which can then be used to augment scores by solving a convex program.  They show that under a convexity assumption on owners' utility functions, the best strategy from owners is to submit honest rankings of their own items. Subject to this behavior, they then show that the Isotonic Mechanism improves accuracy overall.  

The paper introduces a novel perspective to conference peer review, while the author involves in the review process. For an author with multiple papers, the author rates all her papers. Subsequently, the reviewer scores are projected using isotonic regression, subject to the author ranking. If the author has a convex utility over the papers, the optimal action to report the truthful ranking. The paper also considers many extensions, including multiple paper authors, block form ranking, non-separable utility functions, true-score dependent utility, and using soft constraints instead of isotonic regression.