You Are the Best Reviewer of Your Own Papers: An Owner-Assisted Scoring Mechanism
Authors: Weijie J Su
Venue: NeurIPS 2021 Poster (2021)
Keywords: Peer review
mechanism design
ranking
isotonic regression
utility
convex optimization

Abstract: I consider the setting where reviewers offer very noisy scores for a number of items for the selection of high-quality ones (e.g., peer review of large conference proceedings) whereas the owner of these items knows the true underlying scores but prefers not to provide this information. To address this withholding of information, in this paper, I introduce the Isotonic Mechanism, a simple and efficient approach to improving on the imprecise raw scores by leveraging certain information that the owner is incentivized to provide. This mechanism takes as input the ranking of the items from best to worst provided by the owner, in addition to the raw scores provided by the reviewers. It reports adjusted scores for the items by solving a convex optimization problem. Under certain conditions, I show that the owner's optimal strategy is to honestly report the true ranking of the items to her best knowledge in order to maximize the expected utility. Moreover, I prove that the adjusted scores provided by this owner-assisted mechanism are indeed significantly more
accurate than the raw scores provided by the reviewers. This paper concludes with several extensions of the Isotonic Mechanism and some refinements of the mechanism for practical considerations.

You Are the Best Reviewer of Your Own Papers: An Owner-Assisted Scoring Mechanism
Weijie Su Department of Statistics and Data Science University of Pennsylvania suw@wharton. upenn edu
Abstract
consider the setting where reviewers offer very noisy scores for a number items for the selection of high-quality ones (e.g., peer review of large conference proceedings) whereas the owner of these items knows the true underlying scores but prefers provide this information. To address this withholding of information, in this paper, I introduce the Isotonic Mechanism. simple and efficient approach to improving on the imprecise raw scores by leveraging certain information that the owner is incentivized to provide. This mechanism takes input the ranking of the items from best to worst provided by the owner, in addition to the raw scores provided by the reviewers. It reports adjusted scores for the items by solving a convex optimization problem. Under certain conditions, I show that the owner's optimal strategy is to honestly report the true ranking of the items to her best knowledge order to maximize the expected utility. Moreover, prove that the adjusted scores provided by this owner-assisted mechanism are indeed significantly more accurate than the raw scores provided by the reviewers. This paper concludes with several extensions of the Isotonic Mechanism and some refinements of the mechanism for practical considerations.
Introduction
Designing mechanisms for evaluating a set of items by aggregating scores from agents has applications in wide range of domains 2 25 As example of these applications, reviewers asked to score submissions for conference proceedings on the basis of certain criteria, and the scores serve as crucial factor in accepting or rejecting the submissions. Ideally, that is expected to produce scores that adequately reflect the quality of the submissions.
wishes design mechanism
In the last few years, however, the explosion in the number of submissions to many artificial intelligence and machine learning conferences has made it very challenging to maintain high-quality scores from peer review. For instance, the number of submissions to NeurlPS leading machine learning conference-hae has grown from 1,673 2014 to a record-high of 9,454 in 2020 [8] 13] On the other hand, the number of qualified reviewers those, for example, who have published least one paper in a premier machine learning conference conference-simply simply cannot keep up with the growth of the submission size (24. This significant disparity imposes significant burdens on the qualified reviewers, thereby reducing the average amount of time spent on reviewing paper and, worse, pushing program chairs solicit novice reviewers for top-tier machine learning conferences, including NeurlPS, ICML, AAAI, and many others (26
Given the fundamental role of peer review in scientific research, the consequence of this reality has started to impede the advance of machine learning research [17 g 28). In the famous NIPS experiment in 2014 [88, review scores are observed to have surprisingly high level of arbitrariness. The situation has presumably become even worse, as the number of submissions in 2020 was
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

more than five times that when the experiment was conducted. While a moderate level of review arbitrariness reflects the diversity of the scientific community, such a high level as in the 2014 NIPS experiment would very likely lead to acceptance of many low-quality papers and, correspondingly, rejection of a high volume of high-potential papers, In the long term, the poor reliability of peer review might erode the public confidence in these machine learning conference proceedings, challenge that the scientific community cannot afford to ignore [23].
To address this challenge, in this paper I introduce a mechanism that allows one to produce more reliable scores that are modified from the raw review scores. While it seems unrealistic to extend the reviewer pool or assign more papers to each reviewer, it is messible-hhouse rarely done to extract information from the authors and incorporate it into the improvement of scores (see, e.g.. (25)). In fact, an author often submits multiple papers simultancously to the same conference and, moreover, likely to be among the most capable persons to assess the quality of her submissions. Unfortunately, if asked, the author has an incentive to report higher scores than what she believes so as to maximize the utility. To reconcile this dilemma, instead, my mechanism requires the author to report ranking of her submissions in decreasing order of the quality. With this ranking in place, the mechanism solves its adjusted scores from a simple convex programming problem- wwhhh is in the form of isotonic regression 4 using the raw review scores and the ranking as data. Thus, this new mechanism is referred to as the Isotonic Mechanism.
The Isotonic Mechanism comes with several appealing properties that guarantee its superiority over the use of the raw review scores for decision making. First, I show that rational author's 's unique optimal strategy is to report the true ranking of her submissions under the assumption that the utility of the author an additive convex function over all the submissions. Even in the case where the author is not completely certain about the ranking of the "true underlying" scores, the author is better off reporting the most accurate ranking as far as she knows. Second, prove that the Isotonic Mechanism supplied with the optimal strategy taken by the author strictly improves the accuracy of estimating the true underlying scores.
Our further analysis reveals that the gain of the Isotonic Mechanism over the raw scores becomes especially substantial the variability of the scores provided by reviewers is high and the number of submissions by the author is large-the the very situation that many machine learning conferences are faced with. While the high variability often results from the reduction of review time per submission and the arbitrariness of novice reviewers [10], more recently it has become increasingly common for an author to submit multiple papers to one single conference. For example, there are 133 authors who submitted at least five papers to ICLR in 2020, which constitutes significant portion of the total 2,594 submissions. Even more surprisingly, the maximum number of submissions authored by one single individual is as high as 32 in this conference 277. These ever-grooing statistics render the Isotonic Mechanism an ideal approach to improving peer review these scenarios.
In relating to the literature, the Isotonic Mechanism is largely orthogonal to many existing approaches to obtaining better review scores 119 IT 29]. In this line of research, the primary focus develop mechanisms that incentivize reviewers to provide more accurate scores or to sharpen the scores by modeling the generation of scores. In stark contrast, the Isotonic Mechanism solely exploits the information from the authors' side in addition to the raw scores. This orthogonality suggests the promise of incorporating the underlying idea of the Isotonic Mechanism into many existing approaches. For completeness, I remark that an emerging line of research in the mechanism design area leverages additional author information to improve peer review [1 22
2 Methodology
In this section, I introduce the Isotonic Mechanism in detail, and provide theoretical guarantees under certain conditions showing that (1) the owner/author would report her true ranking of her items/submissions in order to maximize the expected utility, and (2) this mechanism improves the overall statistical estimation efficiency for evaluating the papers. Throughout this paper. consider an owner of the items who is fully rational in that her ultimate goal is to maximize the expectation of the overall utility as much as she can.
Consider n items and let their true scores be denoted by RA, Re.... Ra- The reviewers rate the n items y-...r Ba: which given by
R, + 24
(I)
2

---

for i 1...., n. Above, 2 's are the noise terms. In our setup, the owner of the n items is asked to provide ranking of the items, denoted by и. which is permutation of 1, ,...., n. Ideally, the permuted true scores
π R = (Ra()) RT(2)..... Rx(a))
are (approximately) in the nonincreasing order, but it does not necessarily have to be this case, as the owner has complete freedom report any ranking. In our mechanism, the ranking serves shape- restricted constraint that shall facilitate better estimation of the true underlying scores Ri...., Rs based the raw scores y (yI,.... Vn). More precisely, taking as input the raw scores by the reviewers and the ranking ı by the owner, our mechanism outputs scores R == (R1,..., R.,,) that either serve as the adjusted scores or as reference for the decision-making by, for example, the program chair.
Now I introduce the Isotonic Mechanism for adjusting the raw scores by the reviewers, provided the input data y and ranking æ. Throughout the paper, ||
denotes the Euclidean norm.
The Isotonic Mechanism reports R(T), which the optimal solution of the following convex program:
min 2 y rI
s.t. rx(1) (2(2) 2.. 2..--Ta(()) T=(n)-
When clear from the context, omit the dependence on æ when writing the reported ranking R.
In words, the Isotonic Mechanism finds the solution by projecting the raw scores onto the feasible region {r (=(1) 2 (2(2)
2 Tn(n)). which constitutes conver cone that is often referred to as the isotonic cone (up to permutation), hence the name Isotonic Mechanism. Needless to say, the mechanism wishes that the owner can report the true ranking, although is not necessarily the case. If the reported ranking is T², the true values R is feasible point of the program (2) and this program is essentially a restricted least-squares estimator. The projection shall presumably improve the estimation accuracy as it greatly reduces the search space for the true scores.
From algorithmic angle, an appealing feature of this mechanism is that is convex quadratic programming problem and, therefore, is tractable. Even better, this program admits a certain structure that allows for very efficient algorithm called the pool adjacent violators algorithm [16] 6] Interested readers are referred to [2] for other algorithms for solving isotonic regression.
To proceed, several pressing questions concerning the Isotonic Mechanism must be addressed. What is the optimal strategy of the owner and will she report the true ranking out of her own interest? Morcover, does the mechanism, provided that the owner chooses the optimal strategy, really outperform the raw scores of the reviewers in any sense? These questions will be addressed by the following subsections.
2.1 Optimality
To find the optimal strategy of the owner, first recognize that the owner is rational and aims to maximize her expected utility by supplying any ranking of her choice the Isotonic Mechanism ( On the surface, it seems unclear which ranking would maximize the expected utility.
I make the following assumptions for the setup for the mechanism. Before introducing the assump- tions, it is worth noting that the next section extends our results to a certain relaxed version of these assumptions.
Assumption 2.1. Given (final) scores Ri...., Rr of n items in the possession of the owner, then the owner's utility takes the form Util( R. = EE= U(R.). where is a nondecreasing convex function.
In this assumption, the utility function is separable in the sense that it is additive over all the items. Regarding the convexity assumption of the utility function, note that it boils down to requiring that the marginal utility (r) (assume differentiability) is not consistent with the conventional law of diminishing marginal utility [15]. However, a more
nondecreasing function. At first glance,
careful look first shows that score R, does not measure the quantity of some matter. Morcover, note
3

---

for any that for peer review in conference proceedings, a high score largely determines whether the paper would be presented a poster, oral talk, or even as a plenary talk, which really makes significant difference for paper in terms of impact. In this spirit, the urility increases rapidly as the score increases and therefore a convexity assumption on the utility sheds light on this reality.
In Section 3 consider an extension of this assumption. In addition, I discuss this assumption in the practical context of peer review in Section T
Notably, the utility in many applications can vanish to zero once the score r is below some value. In view of this fact, examples of such utility functions include (rr max ar + b} and Ur) max (0, ear+b c). where and are positive constants.
Assumption 2.2. The owner has knowledge of the true ranking of her items. That is, the owner knows which permutation x* that makes π²" o R in nonincreasing order.
Remark 2.1. 1. Turning to Assumption 2.2 it is worthwhile mentioning that the owner only needs to compare the true values instead of knowing the exact values of R in order to report the true ranking. In Section 3 I relax this assumption by considering owner who possesses only partial knowledge of the true ranking.
Assumption 2.3. The noise (21,..., =n) follows an exchangeable distribution in the sense that (2))...,Zn) has the same probability distribution in RK as p permutation p of I,..,,
Remark 2.2. This assumption includes the often-assumed setting where (..,..., 2n) is comprised of independent and identically distributed random variables. This generality is useful in the case where the noise terms involve common latent factor- for example, a recent trend in research directions- invalidates independence but the exchangeability remains true. Furthermore, is beneficial to note that if the score of the i-th paper is averaged over several reviewers, the noise terms may have different variances. However, our assumption remains valid if can treat the number of reviewers assigned to each paper as an independent and identically distributed random variable. In addition, technically speaking, our assumptions do not require the noise to have zero mean, though I do not intend emphasize this generalization.
Interestingly, the following theorem shows that reporting the true ranking is just the optimal strategy. Recall that the true ranking n² is a permutation of 1, ......P n such that the permuted true scores obey (1)(1) 2 (2) 2 ... 2 (x)( and the expected utility
E Util( R = E
U(3))
(3)
taken over the randomness of the noise terms
Theorem 1. Under Assumptions 2.3 the expected utility 3 is maximized when the Isotonic Mechanism is provided with the true ranking x². That is, the owner's 's optimal strategy is to honestly report the true ranking.
To better appreciate this theorem, note that this optimal strategy is "computationally free" in the sense that reporting the true ranking does not require further effort from the owner under our assumptions. In particular, the optimal strategy does not rely on the raw scores provided by the reviewers. Moreover, as revealed by the proof in Section 2.2 if the true scores R. do not contain ties and the function U is strictly convex, the optimality claimed in Theorem T is strict in the sense that E Util( R(T)) < E Util( [R(T")) for any ranking ı that is not identical to π.. The gap between the two utility terms can be arbitrarily large when the true underlying scores are widely different from each other. Even in the presence of ties, the inequality above remains strict unless π is also a true ranking, thereby only differing from "' in the indices corresponding to the ties.
In light of Theorem in the remainder of this section I assume that the owner reports the true ranking out of self-interest. The next step is study the performance of the Isotonic Mechanism from social welfare viewpoint. Explicitly, for example, are the adjusted scores more accurate than the raw scores? The following theorem answers this question in the affirmative.
Theorem 2. Under Assumptions 2. 2.3 the Isotonic Mechanism improves the estimation accuracy of the true underlying scores in the sense that
2
a
E Σ
(R((=T) R,
< E
(yi
R2)²
4

---

Remark 2.3. Above, the l2 loss is chosen mainly for technical convenience. An interesting question for future research is to analyze the performance using other loss functions.
Together with Theorem I Theorem 2 demonstrates the superiority of the Isotonic Mechanism over the use of the raw scores by the reviewers. Furthermore, the following result shows that the gained improvement becomes more significant in the case when the number of items and the noise variance are large.
To state this theorem, below I impose more refined assumptions on the noise terms. Besides, I let
n-1
V(R) = Σ (1+1) - -Ra+ = max R4 min R,
denote the total variation of R.
Theorem 3. In additions to Assumptions 2.2 and 2.1 assume that 21. 2n are independent and identically distributed normal random variables N (0, o²) Letting both and V >> 0 be fixed. I have
( E Σ (Ri(TT)) R.) R
0.4096 + (1) Σ
- 7.5625 + on,(1,
where both on (1) +0
Remark 2.4. This theorem is adapted from well-known result of isotonic regression (see, e.g.. Theorem 2.3 in (30)) and, thus the proof is omitted. As an aside, the risk bounds should not be interpreted as being always increasing with the total variation bound V. To see this, note that when (++1) >> Rg)(() for all i, then the raw scores yi.....> Yn satisfy the isotonic constraint (2) with ı set to π with high probability. As consequence, the risk of the Isotonic Mechanism roughly equal to that of the raw scores. Even if the total variation V tends to infinity, the risk stays constant.
Loosely speaking, this theorem says that the risk of the Isotonic Mechanism is of order 0(n1/34/33)
For comparison, note that the risk of the raw scores is E Σ (y: R,)² E -> = no²
Recognizing their dependence on the noise level a, see that the Isotonic Mechanism especially preferable when is large or, put differently, when the reviewers give very noisy scores. Moreover, the Isotonic Mechanism has a risk that grows much slower with respect to the number of items n.
2.2 Sketch of Proof
This section is devoted to proving Theorem 1 and we defer the proof of Theorem We need the following two lemmas for the proof of Theorem [
2 to the Appendix.
Lemma 2.4. Let x: (21...., Fn) Ž y (.1...., Un) in the sense that #1 2 D1. 21 + I2 > y1, + v2...., 21 + +.... In-1 2 31 +.... yn-1 and x1 + ..... + Fn = 21 + +
+ yn- Let x+ and rn). respectively.
yy be the projections of and y onto the isotonic cone {r Then, we have x+ ž yf.
r2
Lemma 2.5 (Hardy-Littlewond-Pooe
inequality). Let be convex function. Assume that x and y are nonincreasing vectors (that is, Iı 2 ... 2 Fu and y1 2 ... 2 yn) and x ž y. Then, we have Er fxx) 2 Σ" f(:).
Lemma 2.5 is a well-known result in the theory of majorization. See its proof in 20
In contrast, Lemma 2.4 is new to the literature and its proof requires some novel elements. This partly because the definition of majorization in this lemma is slightly different from that in the literature [20]. In the literature, ž y if the condition in Lemma 2.4hoold after ordering both æ, from the largest to the smallest. To this end, we need the following definition and Lemma 2.7
Definition 2.6. We call 21 an upward swap of 2² if there exists for all k * j and z! + z) = =² + +2 2 2,?
< < such that 2 2
This definition amounts to saying that =1 is upward swap of z2 if the former can be derived by transporting some mass from the entry z² to an earlier entry. easy to check that 11 2 2 if z is an upward swap of z².
5

---

Lemma 2.7. Let x: 2 y. Then there exists an integer m and z z'...., x such that x' x" and z' is an upward swap of 2 for I....,
Due to space constraints, relegate the proofs of Lemma 2.7 and Lemma 2.4 to the Appendix.
Proof of Theorem ! Isotonic Mechanism is
Without loss of generality, assume that R2 2 R2 Rg In this case, the
min y r1?
s.t. rı 2 r2 2 2 rn.
where y = R + 2. is clear, the solution, denoted R, is obtained by projecting onto the isotonic
cone C' = {r 2 r2 rn). Now consider that the owner proposes a different ranking, x, which corresponds to
min 2 y r||
s.t.
Tx(1) 2 (n(2) 2 ... 2 Pa(n).
which is equivalent to
min
2 2 o y r|2
S.t. r1 2
nm
after an appropriate permutation operation. In this case, the owner's ratings be can obtained by projecting onto the isotonic cone, and then pulling back the original permutation by applying Now, defining a as π
az(z)) for a vector a, we observe that ı R + z. For simplicity, we let Projc denote the projection operator. Consider the two projections associated with the two rankings, namely Proje and π Projo y). Due to our construction of the utility function, the owner is indifferent between
(az(1).
Projc y) and (Tojc Next, making use of our assumption on the noise distribution, we have T R++ o o R z. Now we compare Projc + z) and Proje( R 2) in terms of majorization. Using the simple but important fact that R + ž π o R + 2, it follows from Lemma 2.4 that Proje (R + 2) = Projo + +)).
Therefore, Lemma 2.5 gives
Σ U (P 2
Σ
U (Projo( R+ +)),
from which we readily get
U (Projo(R +22)
2 E
Σ U (Projc( R R+ 2)))
R)))))
That is, we have Utilmme makkng
2 Util, for an arbitrary ranking n. This finishes the proof.
3 Extensions
In this section, extend our main results by relaxing the assumptions made in Section 2.1 Our primary aim is to show that the rational owner would continue to report the true ranking her items in more general settings. Omitted proof's are deferred to the Appendix.
Ranking in a block form. Suppose that the owner only knows partial information the true ranking in block sense: let ..,..... 7.m be positive integers satisfying +... + 7. and {1, (1.2..... n
be partitioned into {1, 2, ... n) = 1 U 15 U... 1t, such that [4] = n; for I,..., m. The
owner knows the true ranking satisfies
R R1; R
(4)

---

but the ranking within each block is completely unknown to the owner The owner is asked to report an (ordered) partition I,, .i..... In of {1...., n such that [,] n, for i l..... m. Now we
consider the following ,,,, mm (m.) >m)block Isotonic Mechanism:
min 2 y r1
(5)
T1, rlg 2 Pr...
Note that it is also convex optimization program.
Assumption 3.1. The owner has knowledge of the true (n1,.... Dm ) --locc ranking of her items. That is, the owner knows which (R1,...; Rmm) (om--sized partition that satisfies 4
Theorem 4. Under Assumptions 2.1 and 2.3 the expected utility 3 is maximized when the
block Isotonic Mechanism is provided with the true (ri1..... -mm --ized (ondered) partition.
Remark 3.1. The theorem above can intuitively be extended in hierarchical manner. That is, for each block 4t the owner can further provide ranking within this block either in a partial sense or complete sense. The theorem shall continue to be valid and a rigorous treatment of this extension is left for future research.
Denote by := (I,,..,, I. Let πTy be formed by padding the m blocks while setting the en- tries of y within each block in nonincreasing order. That is, the set {TI.y (1),....... (T.) = (mr) and likewise for the following m 1 blocks. For example, let n 4,4 11 = {2,4]. and y (4.4, 6.6, 5, -1), then (I), (1) TI = (3, 1, 2, 4) and ysu. = (5, 4.4, 6.6.-1).
I and (1) 2 (2) 2
Robustness to inconsistencies. Next, we extend Theorem | to the setting where the owner might give a ranking that is not consistent with the true values. Imagine that the owner is faced with the problem of choosing between two rankings πτ, T2. While и nor πΣ might not render the true values R in nonincreasing order, we assume that the former is more faithful with respect to R than the latter in the sense that
T1 R = T2 o R.
(6)
To better understand when this condition holds, we say that क is an upward shuffle of m2 if there exists two indices 1 < iie < S n such that
(x)(1) Raz()) Rx,(5) R1y(1)
and (k) (2(k) for all k * i.j. As is clear, (m1, satisfies (6) In general, 6 is satisfied if
Ti can be derived by sequentially upwardly shuffling from π2, by recognizing the transitivity of
majorization. However, it is worth noting that it is not correct that any pair (TI, m2) satisfying (6) (100, 0, 10) and
can always be sequentially shuffled from one to another. An example is á
T2 R (10, 1, 0, 100).
The following result demonstrates that Theorem I Textends this fault-tolerant setting.
Theorem 5. Under Assumptions 2.7 and 2.3 the owner in the face of choosing between Ti and T2
satisfying (6). which known to the owner, would be better off reporting лı rather than T2 in terms of the overall utility.
Non-separable utility functions. A function f TR. + called a Schur-convex function if symmetrie in its n coordinates and f(x) > f(y) for any æ, y such that Iı >... > Ins 91 > ... Yn and æ > y as defined in Lemma 2.4, As shown by the proofs in Section 2.2, the utility Util(r) given in Assumption 2. is Schur-convex. More generally, when f is differentiable and symmetric, then f is Schur-convex if and only if
(0((r) Ör:
as(r) Ör,
ry)
2
for all r (ri,...,, rn) (see, e.g., 19
Recognizing that the final step in the proof of Theorem I relies on the majorization property, we readily obtain the following result.
Proposition 3.2. In addition to Assumptions 2.2 and 2.3 assume that the function Util(r) is Schur- convex. Then, the owner's 's optimal strategy to honestly report the true ranking of her items.
7

---

True score-dependent utility. Intuitively, the utility of each items shall depend on the true score of the item. An owner may value more about the acceptance of a high-qualit, item, In this spirit, we consider utility function that takes the following form:
Util R2) - Σ (R& Ra),
(7)
1=1
where we assume that (r; R) is convex is in its first argument and satisfies
(U(;;
Ör 2 @U R')
Ör
whenever R > R'. For example, consider (r; R) = ((r) where g 2 0 is convex and h 2 0 is a nondecreasing differential function.
The following result shows that Theorem remains valid under this generalization.
Proposition 3.3. In addition to Assumptions 2.2 and 2.3 assume that the function Util(r) given in 7 Then, the owner's 's optimal strategy is to honestly report the true ranking of her items.
The proof of this proposition is basically the same that of Theorem except for a minor modification. Explicitly, observe that
Proof of Proposition
U RR: R.)
is always upper bounded by
Σ U URi Ro(s)
for a permutation o such that R,'s and Rp() 's have the same ranking in decreasing order. An important fact is that in reporting the true ranking, the utility has already been maximized over all permutations. The remaining part of the proof the same as that of Theorem
Soft constraints. Conceivably, one would argue that the hard constraints provided by the isotonic constraint seems too strong, especially in the case where the owner might not be completely sure about the ranking of her items. In this case, a simple adjustment to consider the following convex combination candidate of scores:
: (R(T') (1 0)y,
where 0 0 1. Owing to the convexity of the risk, Theorem 2 remains correct. Proposition 3.4. Under the assumptions Theorem we have
2
E
R
R
<
E
(y)
R4)
Note that this generalization does not change the mechanism in that the owner still reports the ranking in view of the isotonic regression. More generally, we consider the following optimization program:
min
y r| Pen(r),
where the (nonnegative) penalty term Pen(r) encourages isotonic monotonicity in the sense that, for example, Pen(r) if is in the isotonic cone C' and otherwise Pen(r) > 0. A choice of this penalty is
Pen(r) = = (rx(1+1) - Tx())++
where À >> O, is the ranking reported by the owner and x4 max (2, 0}.
As X - oo, the penalty of the violation of the isotonic constraint tends infinity. Therefore, we have the following simple fact.

---

Proposition 3.5. The Isotonic Mechanism can be (asymptotically) recovered by setting X - oo in
However, the coupling strategy is invalid when it comes to proving for this new mechanism. While can we prove that Theorem I holds for this X-dependent mechanism when 2, we leave the proof or disproof of the general case n 2 3 for future research.
Proposition 3.6. When
2, the mechanism above is truthopptmmal.
4 Discussion
In this paper, have introduced the Isotonic Mechanism to incentivize the author of a set of papers to report the true ranking, thereby obtaining more accurate scores for decision making. The mechanism is conceptually simple and computationally efficient. Owing to the appealing optimality guarantees of this mechanism, it is likely that the use of this mechanism can some extent alleviate the poor quality of review scores in many large machine learning conferences. Toward the employment of this mechanism, an intermediate step is test the use of this mechanism. For example, conference organizers can require that randomly selected group of authors submit the ranking and use the outcome of the Isotonic Mechanism for this selected group as test.
In closing, propose several directions to tackle practical challenges in applying the Isotonic Mecha- nism. First, the assumption of convex utility, which is critical for the optimality of the mechanism, might not always hold for some authors. As argued Section 2.1 the utility is convex or approxi- mately convex if the author aims for higher scores in order be considered for more visibility and paper awards, beyond the acceptance of her papers. In contrast, if nothing more than acceptance or rejection makes difference the author, one may turn to thresholded utility function of the form
U 0, if < Yo: u,
if
2 ro-
where u and ro some constants. An important question for future work is, therefore, extend the Isotonic Mechanism in the face of non-convex utility.
From a practical standpoint, another direction is to incorporate strategic behaviors of both sides of the author and reviewers. The author might deliberately submit additional very low-quality papers and rank these as the worst. In this case, the Isotonic Mechanism tends to push upward the scores for "normal" papers, thereby increasing the chance of acceptance for these papers which the author cares about. A possible approach to overcoming this issue is to set screening step to filter very low-quality submissions. Moreover, the Isotonic Mechanism enables the reviewers to have an influence on the author's ss other papers that are examined by different set of reviewers. It is interesting investigate how the reviewers would act for self-interest and improve on the mechanism to confront any violation of ethical codes.
Third, while Theorem 3 shows the great benefits of using the Isotonic Mechanism, it would be of interest to perform experiments to quantify the magnitude of the benefits. As of finishing this paper, the NeurlPS 2021 Program Chairs asked each author to "rank their papers in terms their own perception of the papers' scientific contributions to the NeurlPS community," though the data was not used for making decisions. Nevertheless, an interesting empirical study is to investigate what would be the outcome if the Isotonic Mechanism were used for NeurIPS 2021.
Last, a critical advance is to extend the Isotonic Mechanism to the setting where each item/paper is owned/authored by multiple owners/authors. This is necessary step toward applying this mechanism to large machine learning conferences, where most papers are written by multiple authors. Simply invoking the Isotonic Mechanism for each author does not seem to be a good solution. For example, an author can adversarially rank a good paper co-authored by, say Bob, as the lowest so as to put her papers that are not co-authored by Bob in advantageous position. From different angle, would the use of the Isotonic Mechanism in an appropriate form discourage guest authorship? leave this challenging extension for future work.
Limitations and societal impacts. A clear limitation of our work lies in its assumptions which our theoretical results rely on. Given the pivotal role of peer review in conference proceedings and the growing impact of Al on our society, we believe that the Isotonic Mechanism would have substantial socictal impacts, provided that it would be used someday.

---

Acknowledgments and Disclosure of Funding
I am grateful Nihar Shah and Haifeng Xu for very helpful and enlightening discussions. I would also like to thank the anonymous reviewers for their constructive comments that helped improve the presentation of this work. This work was supported in part by NSF through CAREER DMS-1847415 and CCF-1934877
Alfred Sloan Research Fellowship, and the Wharton Dean's Research Fund.
References
H. Aziz, O. Lev, N. Mattei, S. Rosenschein, and T. Walsh. Strategyproof peer selection using randomiza- tion, partitioning, and apportionment. Artificial Intelligence, 275:295-3099 2019.
Y. Babichenko, Dean, and M. Tennenholtz. cooppiivecoompatibe selection mechanisms for forests. In Proceedings of the 21st ACM Conference on Economics and Computation, pages 111-131, 2020.
[3] R. E. Barlow, D. Bartholomew, J.-M. Bremner, and H. Brunk. Statistical inference under order restrictions: The theory and application of isotonic regression. Wiley New York, 1972.
[4] R. E. Barlow and H. D. Brunk. The isotonic regression problem and its dual. Journal of the American Statistical Association, 67(337):140 -47,777,, 1972.
[5] M. J. Best and N. Chakravarti. Active set algorithms for isotonic regression; unifying framework. 1990.
Mathematical Progranmiing 47(1-3)::25
[6] M. Bogdan, E. Van Den Berg, C. Sabatti, W. Su, and E. J. Candès. SLOPP--daptive variable selection via convex optimization. The Annals Applied Statisticx, 93)::1033-1144 2015.
[7] D. C. Brabham. Crowdsourcing. Mit Press, 2013.
[8] E. S. Brezis and A. Birukou. Arbitrariness in the peer review process. Scientometrics, 123(1):393 411, 2020.
N. Elezović and Pečarić. A note Schur-convex functions. The Rocky Mountain Journal of Mathematics. pages 853-856, 2000.
[10] O. François. Arbitrariness of peer review: bayesian analysis of the NIPS experiment. arXiv preprint arXiv: 15w.1007.0641,, 2015.
[11] T. Gneiting and A. E. Raftery. Strietly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):59--378 2007.
[12] S. Grotzinger and C. Witzgall. Projections onto order simplexes. Applied Mathematics and Optimization, 12(1):247-2700 1984.
[13] S. Ivanov. nizations, and countries.
NeurIPS 2020.
comprehensive analysis of authors, orga-
http:///wdu.ccc////m//m wwt.c/occ//coeeeeeeeeeeeeeeeeeee criteo- wtt.m/rrtteonineirinnn.e...
1iiippp-222--ccopreeeeeeeee www.eeeaaaaaaaalllaaaaaaaae of www.aaoo-ooooggnaatiioooo and- cod.ooooerrreee--1bb0003133 2020. [Online].
[14] S. Jecmen, Zhang. Liu, N. Shah, V. Conitzer, and Fang. Mitigating manipulation peer review via randomized reviewer assignments. In NewrIPS, 2020.
[15] D. M. Kreps. A course in microeconomic theory. Princcton University Press, 1990.
[16] B. Kruskal. Nonmetric multidimensional scaling: numerical method. Psychometrika, 29(2):1 15-129, 1964.
[17] Langford and M. Guzdial. The arbitrariness reviews, and advice for school administrators. Communi- cations of the ACM, 58(4):12-13, 12-13, 2015.
[18] N. Lawrence and C. Cortes. The NIPS experiment, 2014.
[19] Y. Liu, J. Wang. and Y. Chen. Surrogate scoring rules. In Proceedings of the 21st ACM Conference on Economics and Computation, pages 853-871, 2020.
[20] A. W. Marshall, I. Olkin, and B. C. Arnol. Inequalities: theory of majorization and its applications. volume 143. Springer, 1979.
10

---

[21] N. Mattei, P. Turriai, and S. Zhydkov. Peemominatiinn: Relaxing exactness for increased accuracy in peer selection. arXiv preprint arXiv:2004. 14939, 2020.
[22] R. Noothigattu, N. Shah, and A. Procaccia. Loss functions, axioms, and peer review. Journal of Artificial Intelligence Research, 70: 10:1481-1515, 2021.
[23] A. Rogers and I. Augenstein. What can we do to improve peer review in NLP? arXiv preprint urXiv:2010.08663, 2020.
[24] D. Sculley, J. Snoek, and A. Wiltschko. Avoiding tragedy of the commons in the peer review process. arXiv preprint arXiv:1901.06246. 2018.
[25] I. Stelmakh, N. B. Shah, and A. Singh. On testing for biases in peer review. arXiv preprint 1oXiv:/91..13111 2019.
[26] I. Stelmakh, N. B. Shah, A. Singh, and H. Daumé III. A novice-reviewer exprreeen to address scarcity of qualified reviewers large conferences. arXiv preprint arXiv:2011.15050, 2020.
[27] S.-H. Sun.
(CLR2020-ppenreviewdaa ICLR2020- ICLR2020-0penRevveeii Openaeieeeataaa 2020. [Online].
[28] A. Tomkins, M. Zhang, and W. D. Heavlin. Reviewer bias in single-versus houble-blind peer review. Proceedings of the National Academy Sciences, 114448):12708-12713, 2017.
[29] J. Wang, I. Stelmakh, Y. Wei, and N. B. Shah. Debiasing evaluations that are biased by evaluations. arXiv preprint arXiv:20/2.00714 2020.
[30] C. C. Zhang. Risk bounds in isotonic regression. The Annals of Statistics, 30(2):528-53 555, 2002.


Reviews:
[{"review_id": "phA1lSzXIUX", "reviewer": "Reviewer_a48h", "summary": "This paper introduces the Isotonic Mechanism, which uses the ranking data from authors to get rid of the noisy scores from reviewers. Theoretically, if using the proposed mechanism, the best strategy is to report the correct ranking, which justifies the correctness of the proposed mechanism. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the proposed mechanism is interesting and seems very easy to be implemented. The main theorem guarantees that the authors will truthfully report the ranking. The presentation of this paper is also great. The main message is easy to understand, and all theorems are followed with explanations. The problem solved in this paper is indeed critical. Sometimes I also got frustrated by the quality of some reviewers. Theorem 3 is great! It says that the risk can be reduced significantly when $n$ is large.\n\nI have the following three doubts about this paper:\n1. The assumption about the utility function might be too strong. For example, if I am the first author for an okay paper and the fourth author for a good paper.  The optimal strategy of mine might be ranking the okay paper on top of the good paper.  Not sure whether the idea of Isotonic Mechanism can also solve this problem if the author ranking is taken into account.\n2. I am a bit doubtful about the performance of Algorithm 1 in real-world applications. It might be possible that many owners only have one paper, which makes Isotonic Mechanism reduce to the standard mechanism.\n3. I am not very familiar with the related works. Is the author-wise ranking-based approach new in the review system? I think the authors need to provide a clear answer to this question.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "A theoretical paper. No big potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "oNouv4isMp0", "reviewer": "Reviewer_3cVW", "summary": "The author studies the problem of accurate evaluation of papers via peer review. They propose a mechanism by which authors may submit rankings of their submissions ordered by quality, which can then be used to augment scores by solving a convex program.\n\nThey show that under a convexity assumption on owners' utility functions, the best strategy from owners is to submit honest rankings of their own items. Subject to this behavior, they then show that the Isotonic Mechanism improves accuracy overall. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "First of all, I recognize the irony of peer reviewing a paper about (improving noisy and bad) peer review. I would like to say that I really think this is an important problem, as reviewing for conferences is a notoriously noisy process, and the author's proposed mechanism is very interesting. Overall, incentivizing authors to truthfully rank their papers (and perhaps other papers...?) could indeed be useful. \n\nI found the paper interesting and and well-presented, but I have some qualms about the (rather strong) assumptions made in the paper.\n\nI found the convexity assumption throughout the paper to be quite strong and perhaps unrealistic. It seems that some authors may indeed reap diminishing returns from more papers being accepted; at least anecdotally, this seems to be a bigger driver than increasing visibility of more papers. I would be interested to see an extension of the theory in the paper to non-convex (perhaps submodular?) functions.\n\nWith respect to the distribution of papers per author at conferences, it would also be very enlightening to see a complete distribution of the number of papers per author for large conferences like NeurIPS and ICML. I would hazard a guess that the majority of papers are submitted by authors who only have one paper, and thus there will be a large fraction of papers with no additional information about them. Perhaps this will disadvantage the authors with fewer submissions. One other suggestion that could be very interesting would be to allow authors to evaluate their own papers in comparison with other submissions, but to do this in a way such that their own paper's ranking is not affected by their report, a la notions of impartiality in the ranking literature.\n\nAlso, if all authors submit exactly two papers and there is a nice partition of the papers into pairs such that each paper is compared to exactly one other paper, how much does this help increase the accuracy of reviews? It seems like most of the theoretical results really kick in with noisy reviews and many papers per author.\n\nIt also seems like the author is perhaps \"leaving some money on the table\" with respect to multiple owners. If instead of a partition, they allowed for an overlapping collection of subsets where each author with multiple papers was allowed to rank her own papers, this could yield much richer information across subsets that (if everyone reported accurately and honestly) could help the mechanism considerably. Have you considered this option? \n\nAlso, I think that this mechanism could be readily abused by reviewing rings. For instance, if an author has multiple submissions and knows that her strongest paper is being reviewed by people in her reviewing ring, then she can report that it is her weakest paper and presumably benefit from the highly positive reviews given to her now-\"weakest\" paper.\n\n\nMinor comments:\n\nIn Thm 3, it would be nice if you formally defined risk. \n\n224: Theorem 1, Theorem 2\n351-2 repeats 348-9", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "a2bmZOOfcjS", "reviewer": "Reviewer_cAEK", "summary": "This paper looks at the question improving the process of peer review by allowing authors to submit a ranking of their own papers. The idea is to use this ranking from the authors to improve the noisy rankings coming from the reviewers. The authors show that one can formulate this problem as an isotonic regression problem and that if the authors are utility maxamizers for the sum utility of a set of papers then it is rational to submit and hones ranking of the papers. Extensions are also proposed including authors submitting bucket rankings as well as variants of utility functions.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well written and addresses and important topic, namely how to do more with the available reviewing resources at large conferences. The proposed method is interesting and contains some good ideas for one way to achieve this goal: namely though the use of self assessment.\n\nHowever I have several issues with the paper: (1) it is not clear that it is within scope for a computing conference as there is no learning and even the optimization algorithms are not detailed in full. (2) The model as it is currently established seems inadequate to deal withe full complexity of the problem. And (3) there is significant related work that is missing, specifically the large literature on impartial peer review under noise models.\n\nI'll address each of these shortcomings in turn.\n\n*Relevance:* The paper proposes an interesting model that incorporates information in a unique and creative way. However, it is unclear to me that this paper fits within the conference call as it does not address learning, optimization, or even computation in a direct way. Optimization algorithms are left to a reference and despite the availability of real world conference data (e.g., PrefLib or the ICLR data) there no empirical or numerical experiments. To be more precise: while it is true that none of these datasets have self reported rankings one could either run purely numerical experiments, e.g., draw from a general distribution such as a Mallows model and then run the suggested algorithm or one could impute the required orders and draw samples from the existing datasets. All the papers I suggested below have experiments with various adaptations such as this and given the bounds I would have liked to see evidence that this proposed method performs well in practice for *some* set of assumptions.\n\n*Model Adequacy:* The model detailed in the paper assumes that authors have a non-decreasing utility function over their own papers. This is a fundamental assumption of the model and the stragegyproofness of the mechanism rests firmly on this assumption. I also find this assumption to be completely inadequate. It seems more reasonable that there are only two outcomes: accept and reject, and I have a 0/1 preference over this. The paper dismisses this model out of hand, which I find strange. \n\nAdditionally, the noise model is one of random noise, which seems reasonable, but in the literature it seems much more reasonable to use e.g., Mallows models or some kind of Random Utility Model to model the noise in these rankings. I would have liked to see various noise models discussed or at least cited (see below) since there are several that are widely used. \n\nOn line 88 it seems that reviewers have to score all possible papers? This seems completely unfeasible and goes against the arguments above. In all the work in this space (below) we assume some small set of rankings and it's not clear to me how this model would work without the assumption of full utility function elicitation (this is made worse by the lack of computational experiments noted above). \n\nFinally, while it is true that some authors submit many many papers (more than they can possible read) this is being addressed through limits in the number of submissions at many conferences. While you provide stats that many authors provide many papers, it seems that (from the given stats) many more authors have only one paper or perhaps two. I would like to see some discussion of what happens when many of the self rankings are singletons, but this very important case is not discussed.\n\n*Related Work:* I include a few references here but there is significant work in the mechanism design and computational social choice areas that deal with the problem of impartial peer evaluation and review. While the paper current includes references to some work on de-biasing evaluations the much more relevant work on mechanism design (which is what this paper is) is missing. While I think the algorithms in this paper are new, this oversight is significant and worrying.\n\nAziz, H., Lev, O., Mattei, N., Rosenschein, J.S. and Walsh, T., 2019. Strategyproof peer selection using randomization, partitioning, and apportionment. Artificial Intelligence, 275, pp.295-309.\n\nNoothigattu, R., Shah, N. and Procaccia, A., 2021. Loss functions, axioms, and peer review. Journal of Artificial Intelligence Research, 70, pp.1481-1515.\n\nJecmen, S., Zhang, H., Liu, R., Shah, N., Conitzer, V. and Fang, F., 2020. Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments. Advances in Neural Information Processing Systems, 33, pp.12533-12545.\n\nMattei, N., Turrini, P. and Zhydkov, S., PEERNOMINATION: Relaxing Exactness for Increased Accuracy in Peer Selection. IJCAI 2020.\n\nIn all this paper contains a nice idea: use self reporting data in a clever way to improve peer review. However, in its current form it does not do enough to show that this method is applicable to the real problem or demonstrate effectiveness of the method. Finally important related work is missing.\n\n-----\nReview revision after discussion and rebuttal.\n\nThe authors have answered many of my questions on the paper and I have raised my score on this submission. I am now convinced the paper is in scope for the conference but perhaps a few words as to the fit for a machine learning conference are appropriate.\n\nHowever, I still think that a more comprehensive review of the related literature is necessary for this paper -- there are many works in this general area and situating this method is important. Additionally, after discussion I still feel adding synthetic experiments would greatly improve the paper in that it would give some evidence of the quality of improvement of the bound from Theorem 2 since there there is only a \\geq guarantee it would be good to measure this empirically on some imputed data (e.g. a Mallows model). While I agree there will be improvement, some experiments demonstrating this would greatly improve the paper.\n\nIn general I still feel that the model for binary outcomes can and should be investigated in future work as it would greatly expand the scope of the paper's application. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "15UKK4J6cmG", "reviewer": "Reviewer_dEv7", "summary": "The paper introduces a novel perspective to conference peer review, while the author involves in the review process. For an author with multiple papers, the author rates all her papers. Subsequently, the reviewer scores are projected using isotonic regression, subject to the author ranking. If the author has a convex utility over the papers, the optimal action to report the truthful ranking. The paper also considers many extensions, including multiple paper authors, block form ranking, non-separable utility functions, true-score dependent utility, and using soft constraints instead of isotonic regression. \n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea in the paper is very interesting and the theoretical analysis is comprehensive. I think it is a very good paper and I would recommend it for acceptance.\nUsing the author's opinion is something that modern conferences should try.\nAs is also mentioned in the paper, the current results in the paper clearly still have limitations. Here I add a few points/questions in addition:\n\n1. Extension to multiple authors/owners: In the paper, this is proposed in a separate section and no theoretical results are proved. Why is it better to use larger partitions? Can we use multiple owners' rankings to create better rankings? \n\n2. Multiple reviewers: In practice, the area chairs will look at all scores. The current paper only considers one set of scores y (which can be a summary of the reviews). What if we still want to have multiple sets of scores from different reviewers?\n\n3. The choice of isotonic regression and convex utility function is quite important - I think mostly it is because isotonic regression tends to average scores when the ranking is violated. If an author favors high scores over even scores (convex utility) then she should report truthfully. Other combinations of the author's utility and the mechanism are also great future work. \n\nOther minor points:\n\n1. Line 61: There are double \"with the\"\n\n2. Line 290: I think \"fault tolerance\" usually refers to the transformation of input noise (noise of owner ranking) to output noise (the scores R). This section is saying that the owner tends to minimize noise, which is not fault tolerance of the method itself.\n\n3. Line 327: Should be $R_{\\rho(i)}$.\n\n\n----------------\nUpdate: I have read the rebuttal.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The current discussion should be sufficient.", "review_text": "", "extra_scores": {"ethical_concerns": "The current discussion should be sufficient.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Researchers in machine learning and many other fields have regularly complained about the quality of reviews. This paper proposes a novel idea to mitigate the "noise" in the decisions in a peer-reviewed conference. The idea leverages the known fact from statistics and optimization that given a ranking of papers, isotonic optimization (finding the closest answer under the given ranking) can yield significant reduction in noise. The key (very nice) idea in this paper is to ask authors to provide a ranking of their own submissions. The scores given by the reviewers are then projected on these provided rankings. The paper shows that under certain assumptions, authors are incentivized to report their true perceived rankings of their authored papers, which is then expected to yield the noise-reducing benefits of isotonic optimization. 

I have read the paper carefully myself, and the reviewers and I uniformly agree that this is a novel contribution to an important problem. Overall, this paper is a "novel but imperfect" paper. The assumptions here make the current proposal not-yet-practical. Moreover, the paper claims that the assumed conditions are "mild" whereas our assessment is that these assumptions are very strong. (More on this below.) However, I recommend acceptance due to the really fresh perspective offered by this paper to a problem of interest to the NeurIPS community. (I am aware of the typical bias against novel papers in peer review.) 

Interestingly, the paper is also timely given the NeurIPS 2021 experiment of authors reporting the rankings of their papers (although NeurIPS 2021 is not using it for making decisions, unlike the proposal of this paper).

With this acceptance, please ensure the following three action items in the camera ready version:

(A) Please include a thorough discussion in the main text regarding the strong assumptions and resulting challenges.  It is important to discuss them not only to ensure that the reader gets an accurate understanding of the work, but also to facilitate follow-up research towards taking this novel idea to practice. 

(B) Please incorporate the reviewers' comments and the items promised in the rebuttal.

(C) Please see the comments at the bottom of this meta-review regarding *experiments* quantifying the *magnitude* of benefits, and act accordingly.

Based on the reviews as well as my reading of the paper, here is a summary of some issues that arise with the mechanism due to its strong assumptions. 
- - -
1. Convex utilities: This can incentivize reporting of wrong ranking
The paper assumes convex utilities, without any supporting evidence. In my opinion and that of multiple reviewers, the utility for most authors will be quite non-convex with a step-like upward curve near the acceptance threshold. If the utility is non-convex then it incentivizes reporting of the wrong ranking. Here is an example. Suppose I am submitting two papers: one is I think a near-sure accept, and the other is just below bar. Then suppose I rank the worse paper as higher. In this case, under the isotonic method, the top paper's scores will reduce a bit and the worse paper's score will improve a bit. Then my top paper will still get accepted (and hence ranking wrongly didn't really affect me much for this paper) whereas the worse paper has a much higher chance of acceptance under the wrong ranking!

The claim "threshold utilities may not be a sensible choice for modeling the behavior of the user" made in the paper is unsubstantiated and hard to believe. Please remove such claims unless there is supporting evidence.

2. L2 error
It is known that isotonic optimization is good for L2 error. But why is the L2 error a good metric for peer review? In practice, making the accept/reject decision accurately will be considerably important but that is not captured in the L2 error metric analyzed here. Please clarify its purpose in the revision. It is ok to say that this is chosen for theoretical convenience, if that is the case. 

3. Does not account for self-selection, and can incentivize submission of low quality papers
Suppose I have a paper that is below the bar (which in my opinion is worse off than all my other submissions) and I am not planning to submit it in the conventional setting (I don't care about this paper since it is not in good shape yet). However if the isotonic method is employed, then the following happens. I rank this paper as worst. If the ratings received are the lowest then it doesn't affect anything. However, if due to noise, it gets rated higher than other papers, then the isotonic method will increase the rating of my other papers, thereby increasing their chance of acceptance. Hence submitting a low quality paper can help me get my other papers accepted.

4. Not accounting for other strategic behavior: This gives undesirably high power to the reviewer
There are a number of issues of dishonest behavior by reviewers in peer review. One example is where a reviewer doesn't like an author or an author's line of research and so deliberately tries to reject their work. This method will give even more power to this dishonest reviewer. For instance, if this reviewer is reviewing that author's work, then giving a low score to that paper will not only reject this paper but under the isotonic mechanism, can also get other papers by this author rejected. As another example, if there is a reviewing ring, if an author has multiple submissions and knows that her strongest paper is being reviewed by people in her reviewing ring, then she can report that it is her weakest paper and presumably benefit from the highly positive reviews given to her now-"weakest" paper.

5. Exchangibility and authors' stress
Submitting papers is already quite stressful for authors, and the dependence of the decisions on these self reports may significantly add to the stress felt by authors. In particular, it can require authors to estimate the review process, and not just their own paper, which can be stressful.  Also, what exactly should you ask authors? To rank papers in order of their perceived chances of acceptance? Or in terms of their scientific contribution?  These two things need not be the same. The assumption on exchangeability of noise is unnatural, for instance, it is known that novel papers suffer from biases against novelty and interdisciplinary papers also suffer in the peer review process. 

6. "Guest authorship"
The process of publishing in single blind venues has a problem of guest authorship, where a researcher is made an author to give increased visibility or chances of acceptance to the paper even though that researcher has made no real contribution. Will this mechanism further incentivize guest authorship -- somebody who can provide a ranking that can increase chances of a paper getting accepted?
- - -

The authors should revise the paper to make the shortcomings very clear. For instance, the abstract calls the conditions "mild", which does not represent the nature of assumptions accurately, and was disconcerting. Please clarify the nature of the assumptions here and elsewhere in the paper. The exchangability condition on noise is also called "mild" in the paper, which as discussed above, is not. Please expand the discussion section to discuss these issues in the main text. Section 2.3 can be shortened to make room for this. It is important to discuss them not only to ensure that the reader gets an accurate understanding of the work, but also to facilitate follow-up research to make this novel idea practical. 

Minor point: In my opinion, calling it "author-assisted" instead of "owner-assisted"  will be more clear. 

Other points raised in the discussions: Reviewers were concerned about the lack of experiments. I had a number of back-and-forth exchanges with the reviewers on this. We discussed that there is no real data available on authors' perceptions of their own papers and no ground truth on papers to validate. As for synthetic experiments, we discussed that there is a good amount of literature showing the magnitude of benefits of isotonic optimization, but the author should either cite literature empirically quantifying the magnitude of benefits of isotonic optimization (not just orderwise but in terms of actual values) or even better include synthetic simulations in the appendix to illustrate this. There was also some confusion where a reviewer misunderstood that the authors have to rank all submitted papers, and this confusion was clarified in a discussion between the AC and reviewer.
Decision: Accept (Poster)