Counterfactual Evaluation of Peer-Review Assignment Policies
Authors: Martin Saveski
Steven Jecmen
Nihar B Shah
Johan Ugander
Venue: NeurIPS 2023 spotlight (2023)
Keywords: peer review
causal inference
counterfactual policy evaluation
TL;DR: We propose methods for off-policy evaluation in peer review and apply them to data from the TPDP’21 workshop and the AAAI’22 conference.

Abstract: Peer review assignment algorithms aim to match research papers to suitable expert reviewers, working to maximize the quality of the resulting reviews. A key challenge in designing effective assignment policies is evaluating how changes to the assignment algorithm map to changes in review quality. In this work, we leverage recently proposed policies that introduce randomness in peer-review assignment—in order to mitigate fraud—as a valuable opportunity to evaluate counterfactual assignment policies. Specifically, we exploit how such randomized assignments provide a positive probability of observing the reviews of many assignment policies of interest. To address challenges in applying standard off-policy evaluation methods, such as violations of positivity, we introduce novel methods for partial identification based on monotonicity and Lipschitz smoothness assumptions for the mapping between reviewer-paper covariates and outcomes. We apply our methods to peer-review data from two computer science venues: the TPDP'21 workshop (95 papers and 35 reviewers) and the AAAI'22 conference (8,450 papers and 3,145 reviewers). We consider estimates of (i) the effect on review quality when changing weights in the assignment algorithm, e.g., weighting reviewers' bids vs. textual similarity (between the review's past papers and the submission), and (ii) the "cost of randomization", capturing the difference in expected quality between the perturbed and unperturbed optimal match. We find that placing higher weight on text similarity results in higher review quality and that introducing randomization in the reviewer-paper assignment only marginally reduces the review quality. Our methods for partial identification may be of independent interest, while our off-policy approach can likely find use in evaluating a broad class of algorithmic matching systems.

Counterfactual Evaluation of Peer-Review Assignment Policies
Martin Saveski University of Washington msaveski@ur @uw. edu
Steven Jecmen Carnegie Mellon University sj jecmenncss cmu edu
Nihar B. Shah
Johan Ugander
Carnegie Mellon University n niharsCcs. cmu. edu
Stanford University jugander@stannford edu
Abstract
Peer review assignment algorithms aim to match research papers to suitable expert reviewers, working maximize the quality of the resulting reviews. A key chal- lenge in designing effective assignment policies is evaluating how changes to the assignment algorithm map changes in review quality. In this work, leverage recently proposed policies that introduce randomness in peer-review assignment- in order mitigate fraud-as a valuable opportunity to evaluate counterfactual assignment policies. Specifically, we exploit how such randomized assignments provide positive probability of observing the reviews of many assignment poli- cies of interest. To address challenges in applying standard off pollcc evaluation methods, such as violations of positivity, we introduce novel methods for partial identification based on monotonicity and Lipschitz smoothness assumptions for the mapping between reviewer-paper covariates and outcomes. We apply our methods to peer-review data from two computer science venues: the TPDP'21 workshop (95 papers and 35 reviewers) and the AAAI'22 conference (8,450 papers and 3,145 reviewers). We consider estimates of (i) the effect review quality when changing weights in the assignment algorithm, e.g., weighting reviewers' bids vs. textual similarity (between the review's 'ss past papers and the submission), and (ii) the "cost randomization", capturing the difference in expected quality between the perturbed and unperturbed optimal match. We find that placing higher weight on text similarity results in higher review quality and that introducing randomiza- tion in the reviewer-paper assignment only marginally reduces the review quality. Our methods for partial identification may be of independent interest, while our off- off-policy approach can likely find use in evaluating broad class of algorithmic matching systems.
Introduction
The assignment of papers to reviewers one of the most important parts of the peer-reviewed publication process [1-3]. In computer science, conferences are the primary terminal venue for publications, with recent iterations of large conferences such as NeurlPS and AAAI receiving several thousand submissions [4]. As result, these and other large conferences must rely automated systems to decide what members of the impaneled reviewer pool will review each paper. Review matching systems typically use three sources of information: (i) bids, i.c., reviewers' self-reported preferences to review the papers; (ii) text similarity between the paper and the reviewer's publications; and (iii) reviewer- and author-selected subject areas. Given preseribed way to combine these
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

signals into single score, an optimization procedure then proposes reviewer-papp assignment that maximizes the sum of the scores of the assigned pairs [5].
The design of peer-review systems has received considerable research attention [4, 6-8]. Popular peer-review platforms such as OpenReview and Mierosoft CMT offer many features that conference organizers can use to assign reviewers. However, it has been persistently challenging to evaluate how changes to peer-review assignment algorithms affect review quality. An implicit assumption underlying such approaches is that review quality is an increasing function of bid enthusiasm, text similarity, and subject area match, but how to combine these signals into a score is approached via heuristics. Researchers typically observe only the reviews actually assigned by the algorithm and have no way of measuring the quality of reviews under an assignment generated by an alternative algorithm.
One approach to comparing different assignment policies is running A/B tests. Several conferences (NeurIPS' 14 [9, 10], WSDM' 17 [11], ICML'20 [12], and NeurlPS'21 [13] have run A/B tests to evaluate various aspects of their review process, such as differences between single- vs. double-blind review. However, such experiments are extremely costly in the peer review context, with the NeurIPS experiments requiring a significant number of additional reviews, overloading already strained peer review systems. Moreover, A/B tests typically compare only handful of design decisions, while assignment algorithms typically require making many such decisions (Section 2).
In this work, we propose off off-pooliy evaluation less costly alternative that exploits existing randomness to enable the comparison of many alternative policies. Our proposed technique "har- vests" [14] the randomness introduced in peer-review assignments generated by recently-adopted techniques that counteract fraud in peer review. In recent years, in-depth investigations have uncov- ered evidence of rings of colluding reviewers in few computer science conferences [15]. These reviewers conspire to manipulate the paper assignment in order to give positive reviews to the papers of co-conspirators. To mitigate this kind of collusion, conference organizers have adopted various techniques, including a recently introduced randomized assignment algorithm [16]. This algorithm limits the maximum probability (a parameter set by the organizers) of any reviewer getting assigned any particular paper. This randomization thus limits the expected rewards of reviewer collusion at the cost of some reduction in the expected cum-of-similante OpenReview since 2021 and used by several conferences, including 22 and AAAI'
objective, and has been implemented in
The key insight of the present work is that under this randomized assignment policy, range of reviewer-paper pairs other than the exactly optimal assignment become probable to observe. We can then adapt the tools of off-policy evaluation and importance sampling evaluate the quality of many alternative policies. A major challenge, however, is that off-policy evaluation assumes overlap between the on-policy and the off-policy, i.e., that each reviewer-paper assignment that has a positive probability under the off-policy also had a positive probability under the on-policy. In practice, positivity violations are inevitable even when the maximum probability of assigning any reviewer-paper pair is low enough to induce significant randomization, especially as we are interested in evaluating wide range of design choices of the assignment policy. To address this challenge, we build on existing literature for partial identification and propose methods that bound the off-policy estimates while making weak assumptions on how positivity violations arise.
More specifically, we propose two approaches for analysis that rely on different assumptions on the mapping between the covariates (e.g.. bid, text similarity, subject area match) and the outcome (e.g.. review quality) of the reviewer-paper pairs. First, we assume monotonicity in the covariates-outcome mapping. Understood intuitively, this assumption states that if reviewer-paper pair i has higher or equal bid, text similarity. and subject area match than a erviewer-paper pair j. then we assume that the quality of the review for pair is higher or equal to the review for pair j. Alternatively, we assume Lipschitz smoothness in the covariate-outcome idea that two pereee pairs that have similar bids, text similarity, and subject area match, should result in similar review quality. We find that this Lipschitz assumption naturally generalizes so-called Manski bounds [17], the partial identification strategy that assumes only bounded outcomes.
mapping. Intuitively, this assumption captures the
We apply our methods to data collected by two computer science venues that used the recently- introduced randomized assignment policy: the 2021 Workshop on Theory and Practice of Differential Privacy (TPDP) with 95 papers and 35 reviewers, and the 2022 AAAI Conference on Advancement in Artificial Intelligence (AAAI) with 8,450 papers and 3,145 reviewers. We evaluate two design choices: (i) how varying the weights of the bids vs. text similarity vs. subject area match (latter available only in AAAI) affects the overall quality of the reviews, and (ii) the "cost of randomization".

---

i.e., how much the review quality decreased as result of introducing randomness in the assignment. As our measure of assignment quality. we consider the expertise and confidence reported by the reviewers for their assigned papers. We find that our proposed methods for partial identification assuming monotonicity and Lipschitz smoothness significantly reduce the bounds of the estimated review quality off-policy leading to more informative results. Substantively, we find that placing larger weight on text similarity results in higher review quality, and that introducing randomization in the assignment leads to a very small reduction in review quality.
Beyond our contributions to the design and study of peer review systems, the methods proposed in this work should also apply to other matching systems such as recommendation systems [18-20], advertising [21], and ride-sharing assignment systems [22]. Further, our contributions off- policy evaluation under partial identification should be of independent interest.
Our code is available at:
2 Preliminaries
We start by reviewing the fundamentals peer-review assignment algorithms.
Reviewer-Paper Similarity. Consider a peer review scenario with a set of reviewers R and a set of papers P. Standard assignment algorithms for sarge-scale peer review rely on "similarity scores" for every reviewer-paper pair = (r, E R P, representing the assumed quality of review by that reviewer for that paper. These scores S;, typically non-negative real values, are commonly computed from combination of up to three sources of information: (1) text-similarity scores T; between each paper and reviewer's past work, using various techniques [23-28]; (2) overlap K; between the self-nominated subject areas selected by each reviewer and each paper's authors; and (3) reviewer-provided "bids" B, on each paper. Without any principled methodology for evaluating the choice of similarity score, conference organizers manually select a parametric functional form and choose parameter values by spot-checking a few reviewer- reviewer-paper paper assignments. For example, a simple similarity function is convex combination of the component scores: S, = Wtext T2 + (1 Wtext B, Conferences have also used more complex non-linear functions: NeurlPS' 16 [29] used the functional form S, = (0.57; + 0.5K, )2" while AAAI' 1 [6] used S. = (0.52; + 0.5K,) possible functional forms results in wide design space, which we explore in this work.
The range of
Deterministic Assignment. Let Z E {0, 13 || be an assignment matrix where Z, denotes whether the reviewer-paper pair was assigned or not. Given matrix of reviewer-paper similarity |P|] a standard objective is to find an assignment of reviewers to papers that maximizes the sum of similarities of the assigned pairs, subject to constraints that each paper is assigned to an appropriate number of reviewers, each reviewer is assigned no more than a maximum number of papers, and conflicts of interest are respected [5, 27, 30-34). This optimization problem can be formulated as a linear program. We provide detailed formulation in Appendix A. While other objective functions have been proposed [35-37], here we focus on the num-of-similaritics objective.
scores S E 20
Randomized Assignment. As one approach to strategyproofness, Jecmen et al. [16] introduce the idea of using randomization to prevent colluding reviewers and authors from being able to guarantee their assignments. Specifically. the program chairs first choose parameter 9 E [o, 1). Then, the algorithm computes a randomized paper assignment, where the marginal probability P(Z, = 1) of assigning any reviewer-pae peveerrpppprpeeee pair i is at most q. These marginal probabilities are determined by linear program, which maximizes the expected similarity of the assignment subject to the probability limit y (detailed formulation in Appendix A). A reviewer-paper assignment is then sampled using randomized procedure that iteratively redistributes the probability mass placed on each reviewer- wapper pair until all probabilities are either zero one.
Review Quality. The above assignments are chosen based on maximizing the (expected) similarities of assigned reviewer-paper pairs, but those similarities may not be accurate proxies for the quality of review that the reviewer can provide for that paper. In practice, automated similarity- based assignments result in numerous complaints of low-expertise paper assignments from both authors and reviewers [3]. Meanwhile, self-reported assessments of reviewer-paper assignment quality can be collected from the reviewers themselves after the review. Conferences often ask reviewers to score their expertise in the paper's 'ss topic and/or confidence in their review [6, 29, 38). Other indicators of review quality can also be considered; e.g., some conferences ask "meta-reviewers" or other reviewers
3

---

to evaluate the quality of written reviews directly [38, 39]. In this work, we consider self-reported expertise and confidence as our measures of review quality.
3 Off-Policy Evaluation
One attractive property of the randomized assignment described above is that while only one reviewer- paper assignment is sampled and deployed, many other assignments could have been sampled, and those assignments could equally well have been generated by some alternative assignment policy. The positive probability of other assignments allows us to investigate whether alternative assignment policies might have resulted in higher-quality reviews.
Let A be randomized assignment policy with a probability density PA. where ( Pa(Z) 1; (Z(Z) 2 0, vZ; and Pa(Z) > 0 only for feasible assignments Z. Let B be another policy with density PB. defined similarly. We denote by (z(Z,) and Pa( (Z,) the marginal probabilities of assigning reviewer-pappe papprr pair under A and B respectively. Finally, let Y TR where (r; R x P, be the measure of the quality of reviewer r's review of paper p.
We follow the potential outcomes framework for causal inference [40]. Throughout this work, we will let A be the on-policy or the logging policy, i.e., the policy that the review data was collected under, while B will denote one of several alternative policies of interest. In Section we will describe the specific alternative policies we consider in this work. Define N - -iER, x Zi as the total number of reviews, fixed across policies and set ahead time. We are interested in the following estimands:
MA = BA~PA
1 N
YiZ, VERXP
Σ
PB Ez~Pa
1 N
Σ
Y,
iERXP
where yA and jBB are the expected review quality under policy A and B, respectively.
In practice, we do not have access to all Y,, but only those that were assigned. Let Z E {0, 1] RIXPP be the assignment sampled under the on-policy A, drawn from PA. We define the following Horvitz- Thompson estimators of the means:
PA = N
Σ Y.ZAA EERxP
1 PB = N
(n(Z)) Σ Y,Z,W, where W, = (A(Z,) &ERXP
Vi RIER.. (I)
For now, suppose that B has positive probability only where A is positive (also known as satisfying "positivity"): PA(Z,) > 0 for all R P where PB (Z,) > 0. Then, all weights W; where are bounded. As we will see, many policies of interest B go beyond the support of A.
> (n.
Under the positivity assumption, Fa and uin are unbiased estimators of µdd and unn respectively [41]. Moreover, the Horvitz-Thompes estimator is admissible the class of all unbiased estimators [42]. Note that A is the empirical mean of the observed assignment sampled on-policy, and ØB is weighted mean of the observed assignment based on inverse probability weighting. These estimators also rely on standard causal inference assumption of "no interference"; i.e., that the outcomes Y; do not depend on the assignments Z for any other reviewer-pape paper pair j o i. In Appendix B, we discuss the implications of this assumption in the peer review context.
Challenges. In off- off-policy evaluation, we are interested in evaluating a policy B based on data collected under policy A. However, our ability to do so is typically limited to policies where the assignments that would made under B are possible under A. In practice, many interesting policies step outside of the support of A. Outcomes for reviewer-paper pairs outside the support of A but with positive probability under B ("positivity violations") cannot be estimated and must either be imputed by some model or have their contribution to the average outcome (HB) bounded.
In addition positivity violations, we identify three other mechanisms through which missing data with potential confounding may arise in the peer review context: absent reviewers, selective attrition, and manual reassignments. For absent reviewers, i.e., reviewers who have not submitted any reviews, we do not have reason to believe that the reviews are missing due to the quality of the reviewer-pappe paper assignment. Hence, we assume that their reviews are missing at random, and impute them with the weighted mean outcome of the observed reviews. For selective attrition, i.e., when some but not all reviews are completed, we instead employ conservative bounding techniques as for policy-based positivity violations. Finally, reviews might be missing due to manual reassignments by the program

---

chairs, after the assignment has been sampled. As result, the originally assigned reviews will be missing and new reviews will be added. In such cases, we treat removed assignments as attrition (i.e., bounding their contribution) and ignore the newly introduced assignments as they did not arise from any determinable process. Concretely, we partition the reviewer-paper pairs into the following (mutually exclusive and exhaustive) sets:
D I positivity violations, {i = pr E R X PA(Z.) = A PB(Z.) > 0}.
I Abe missing reviews where the reviewer was absent (submitted no reviews).
> IAt;; remaining missing reviews, and
I+: remaining pairs without positivity violations or missing reviews, (RxP) (RXP) In the next section, we present methods for imputing or bounding the contribution of I to the estimate of fıg, and I and I the estimates of ju and P.B.
4 Imputation and Partial Identification
In the previous section, we defined three sets of reviewer-paper pairs for which outcomes Y, must be imputed rather than estimated: I Ibs I In this section, we describe varied methods for imputing these outcomes that rely on different strengths of assumptions, including methods that output point estimates and methods that output lower and upper bounds of ftz- In Section 5, we apply these methods to peer-review data from two computer science venues.
For missing reviews where the reviewer is absent (I-Abs), assume that the reviewer did not partici- pate in the review process for reasons unrelated to the assignment quality (e.g., too busy). Specifically, we assume that the reviewers are missing at random and thus impute the mean outcome among I the pairs with no positivity violations or missing reviews: Y (Eiez+ Y.Z.A.))(Eier) W.W.).
In contrast, for positivity violations (I-) and the remaining missing reviews (IAt)), we allow for the possibility that these reviewer-paper pairs being unobserved correlated with their unobserved outcome. Thus, we consider imputing arbitrary values for in these subsets, which we denote by y-Impute and place into matrix Ylmpute E R. [ leaving entries for i g I UZAt undefined. This strategy corresponds setting Y, Y.!mpute for i I U I in estimator (1). To obtain bounds, we impute both the assumed minimal and maximal values of y!mpput
These modifications result in Horvitz. Morvitz-Thompson off-policy estimator with imputation. To denote this, we redefine ýi to be a function Pa ||||x|P| + I. where p hmpuie) denotes the estimator resulting from imputing entries from particular choice of Y-lputt,
F (Y Impute)
Σ Y,A,W,+ EE
Σ VEA
Y lmpute Z,AW
Σ YZAW, +
Σ lmmuute Pr (Z.)
N
VETAR
1EI
The estimator computes the weighted mean of the observed (Yt) and imputed outcomes (Y-lmpute and Y). We impute y-!mpule for the attrition (I and positivity violation (I pairs, and Y for the absent reviewers (I.bb)). Note that weight the imputed positivity violations (I-) by Ps(Z;) rather than Z, W. since the latter is undefined. Under the assumption that the imputed outcomes are accurate, B (ylmpute) is unbiased estimator of μB. We can further construct confidence intervals using the methods described in Appendix C.
Next, we detail several methods by which we choose Ylmpute, These methods rely on various assumptions of different strength about the unobserved outcomes.
Mean Imputation. As first approach, we assume that the mean outcome within I is representa- tive of the mean outcome among the other pairs. This a strong assumption, since the presence of pair in I or Att may not be independent of their outcome. For example, if reviewers choose not to submit reviews when the assignment quality is poor, Y is not representative of the outcomes in AM Nonetheless, under this strong assumption, we can simply impute the mean outcome for all pairs necessitating imputation. Setting y limute = for all E I AIf we consider the following point estimate of μp: (Y)) While following from an overly strong assumption, we find it useful compare our findings under this assumption findings under subsequent weaker assumptions.
Model Imputation. Instead of simply imputing the mean outcome, we can assume that the un- observed outcomes Y, are some simple function of known covariates X, for each reviewer-paper
5

---

pair i. If so, we can directly estimate this function using a variety of statistical models, resulting in a point estimate of pa. In doing so, we implicitly take on the assumptions made by each model, which determine how to generalize the covariate-outcome mapping from the observed pairs to the unobserved pairs. These assumptions are typically quite strong, since this mapping may be very different between the observed pairs (typically good matches) and unobserved pairs (typically less good matches).
Specifically, given the set of all observed reviewer-paper pairs O = EiE 2,- 1}. =}, we train a model m using the observed data {(,,, Y)) i O}. Let (mm) R [ denote the outcomes predicted by that model for each pair. We then consider jy (Y(m)) as a point estimate of jgg In our experiments, we employ standard methods for classification, ordinal regression, and collaborative filtering: logistic regression (clf-logistic)) ridge classification (clf-ridge); ordered logit (ord- oord-o ordered probit (ond-probit); SVD++ collaborative filtering (cf-svd++); and K-nearest-neighhor col- laborative filtering (cf-knn). Note that, unlike the other methods, the methods based on collaborative filtering model the missing data by using only the observed reviewer-pape parer--pperrae outcomes (Y). We discuss our choice of methods, hyperparameters and implementation details in Appendix E.
Manski Bounds. As a more conservative approach, we can exploit the fact that the outcomes Y; are bounded, letting us bound the mean of the counterfactual policy without making any assumptions on how the positivity violations arise. Such bounds are often called Manski bounds [17] in the econometrics literature on partial identification. To employ Manski bounds, we assume that all outcomes Y can take only values between Umin and Vmax e.g., self-reported expertise and confidence scores are limited to a pre-specified range on the review questionnaire. Then, setting Y Imppte Vmin or y-Impute ymax for all I- U IAM, we can estimate the upper and lower bound of μ* as a (min)) and i (in(ymaz) ). In Appendix C, we discuss how we construct confidence intervals for these estimates that asymptotically contain the true value of uu with probability at least 95% by adapting the inference procedure proposed in [43].
Monotonicity and Lipschitz Smoothness. We now propose two styles of weak assumptions on the covariatc-outcome mapping that can be leveraged to achieve tighter bounds on üB than the Manski bounds. In contrast the strong modeling assumptions used in the mean and model imputation, these assumptions can be more intuitively understood and justified as conservative assumptions given particular choices of covariates.
Monotonicity, The first weak assumption we consider is a monotonicity condition. Intuitively, monotonicity captures the idea that we expect higher expertise for reviewer-paper pair when some covariates are higher, all else equal. For example, in our experiments, we use the similarity component scores (bids, text similarity, subject area match) as covariates. Specifically, for covariate vectors X; and Xj, define the dominance relationship X; > X, to mean that X; is greater than or equal to X, in all components and X; is strictly greater than Xj in at least one component. Then, the monotonicity assumption states that: if X; > X,. then Y. 2 Y,, v(i,j) E (R x P)?
Using this assumption to restrict the range of possible values for the unobserved outcomes, we seek upper and lower bounds on f4B. Recall that O is the set of all observed reviewer-paper pairs. One challenge is that the observed outcomes themselves (Y, for i O) may violate the monotonicity condition. Thus, to find an upper or lower bound, we compute surrogate values Y, E R that satisfy the monotonicity constraint for all i E O UZ Abe UI- while ensuring that the surrogate values Y, for E O are close possible to the outcomes Y, The surrogate values Y, for i E Att UI- can then be imputed as outcomes.
Inspired by isotonic regression [44], we implement a two-level optimization problem. The primary objective minimizes the & distance between Y, and Y, for pairs with observed outcomes E O. The second objective either minimizes (for a lower bound) or maximizes (for an upper bound) the sum of Y, for the unobserved pairs i At y A UI- weighted as in ud. Define the universe of relevant pairs 41 = O U Att U UI- and define as a very large constant. This results in the following

---

pair of optimization problems:
(YM Y-M argmin min! man
V
Σ iE0
|r, - 1//
Σ
Y.W+ + Σ XiPales) SEI
(2)
YIeEul
iEI Abs
Y1 v(i,)) (U2 X, ,,}
Ymin S Umax Vi WiEl U.
Smax
The sign of the second objective term depends on whether a lower (negative) upper (positive) bound being computed. The last set constraints corresponds to the same constraints used to construct the Manski bounds described earlier, which is combined here with monotonicity to jointly constrain the possible outcomes. The above problem can be reformulated and solved as a linear program using standard techniques. We can then estimate the upper and lower bound of μæ as pB min and f (YrMar)) ), and construct 95% confidence intervals as discussed in Appendix C.
Lipschitz Smoothness. The second weak assumption we consider is a Lipschitz smoothness assump- tion on the correspondence between covariates and outcomes. Intuitively, this captures the idea that we expect two reviewer-paper pairs who are very similar in covariate space to have similar expertise. For covariate vectors X; and Xy. define d(X;, X2) as some notion of distance between the covariates. Then, the Lipschitz assumption states that there exists a constant such that Y - Y, I Ld(X;,X)) for all (i, (R x P)2. In practice, we can choose an appropriate value of L by studying the many pairs of observed outcomes in the data (Section 5 and Appendix G), though this approach assumes that the Lipschitz smoothness of the covariate-outcome function is the same for observed and unobserved pairs.
As in the previous section, we introduce surrogate values Y, E R and implement two-level optimization problem to address Lipschitz violations within the observed outcomes (i.e., two observed pairs are very close in covariate space but have different outcomes). The resulting pair of optimization problems is defined exactly as in problem (2), except that the first set of constraints replaced with constraints IY; - Yj & Ld(X,, X,), ((i,)) E U. We include the complete LP formulation in Appendix D. We denote the solutions these problems by Y and Y maz As before, the sign of the second objective term depends on whether lower (negative) or upper (positive) bound is being computed. The last set of constraints the Manski bounds described earlier, which here are combined with the Lipschitz assumption to jointly constrain the possible outcomes. In the limit, as L - co, the Lipschitz constraints become vacuous and we recover the Manski bounds. This problem can again be reformulated and solved as a linear program using standard techniques. We can then estimate the upper and lower bound of μon as B (YL, ) and f (.aa.) and construct 95% confidence intervals as discussed in Appendix C.
again the same constraints used to construct
Experiments
We apply our framework to data from two venues that used randomized paper assignments as described in Section 2, the 2021 Workshop on Theory and Practice of Differential Privacy (TPDP) and the 2022 AAAI Conference on Advancement in Artificial Intelligence (AAA/). In both settings, we aim to understand the effect that changing parameters of the assignment policies would have on review quality. The analyses were approved by our Institutional Review Boards.
Datasets.
TPDP. The TPDP workshop received 95 submissions and had a pool of 35 reviewers. Each paper received exactly 3 reviews, and reviewers were assigned 8 or 9 reviews, for total of 285 assigned reviewer-paper pairs. The reviewers were asked to bid on the papers and could place one of the following bids (the corresponding value of B; shown in the parenthesis): "very low" (-1), "low" (-0.5), "neutral" (0), "high" (0.5), or "very high" (1), with "neutral" as the default. The similarity for each reviewer-paper pair was defined as a weighted sum the bid score, B,, and text- similarity scores, T:: S = Wrext T; + (1 Wtext B,, with Wrext = 0.5. The randomized assignment was run with an upper bound of q = 0.5. In their review, the reviewers were asked to assess the alignment between the paper and their expertise (between 1: irrelevant and 4: very relevant). and to report their review confudence (between educated guess and 5: absolutely certain). We consider these two responses as measures of quality.
7

---

AAAI In the AAAI conference, submissions were assigned to reviewers in multiple sequential stages across two rounds of submissions. We examine the stage of the first round where the randomized assignment algorithm was used to assign all submissions to a pool of "senior reviewers." The assignment involved 8,450 papers and 3, 145 reviewers; each paper was assigned to one reviewer, and each reviewer was assigned at most 3 or 4 papers based on their primary subject area. The similarity St for every reviewer-paper pair i was based on three scores: text-similarity Ti E [0. 1]. subject-area score K2 E [0, 1J. and bid B, Bids were chosen from the following list (with the corresponding value of B, shown in the parenthesis, where Abid = 1 is a parameter scaling the impact of positive bids as compared to neutral//negative bids): "not willing" (0.05), "not entered" (1), "in a pinch" (1+0.5Abaa). "willing" " (1 + 1.5)) "eager" (1 + 3dbsa). The default option was "not entered". Similarities were computed as: S; (Wrext T₁ + (1 (toxtt (P) 1/BB, with Wrext 0.75. The actual similarities differed from this base similarity formula in few special cases (c.g.. missing data); we provide the full description of the similarity computation in Appendix F. The randomized assignment was run with q 0.52. Reviewers reported expertise score (between O: not knowledgeable and 5: expert) and a confidence score (between O: not confident and 4: very confident), which we consider as our quality measures.
Assumption Suitability. For both the monotonicity and Lipschitz assumptions (as well as the model imputations), we work with the covariates X,, vector of the two (TPDP) or three (AAAI) component scores used in the computation of similarities. In Appendix G, we consider whether these assumptions are reasonable with respect to our choice of covariates by evaluating them on the observed outcomes. For the monotonicity assumption, we find that it holds on large majorities of observed outcome pairs. For the Lipschitz assumption, we use a normalized L distance and choose values of L corresponding to less than 10%, 5%, and 1% violations on the observed outcomes.
Results. We now present the analyses of the two datasets using the methods introduced in Section 4. For brevity, we report our analysis using self-reported expertise as the quality measure Y, but include the results using self-reported confidence in Appendix L. When solving the LPs that output alternative randomized assignments (Appendix A), we often encounter multiple unique optimal solutions and employ persistent arbitrary tie- tie-breaking procedure to choose amongst them (Appendix H).
TPDP We perform two analyses on the TPDP data, shown in Figure (left). First, we analyze the choice of how to interpolate between the bids and the text similarity when computing the composite similarity score for each reviewer-paper pair. We examine range of assignments, from an assignment based only on the bids (Wrext = 0) to an assignment based only on the text similarity (Wrext = 1), focusing our off- off-policy evaluation on deterministic assignments (i.e., policies with q 1). Setting Wtext E j0, 0.75] results in very similar assignments, each of which has Manski bounds overlapping with the on-policy (Figure 1A, left). Within this region, the models (Figure IB. left), monotonicity bounds (Figure IC, left), and Lipschitz bounds (Figure ID, left) all agree that the expertise is similar to the on-policy. However, setting Wtext E (0.75, 0.9) results in a significant improvement in average expertise, even without any additional assumptions (Figure A, left). Finally, setting Wtext E (0.9. 1] leads to assignments that are significantly different from the assignments supported by the on-policy, which results in many positivity violations and wider confidence intervals, even under the monotonicity and Lipschitz smoothness assumptions (Figures IA-D, left). Note that within this region, the models significantly disagree on the expertise (Figure IB, left), indicating that the strong assumptions made by such models may not be accurate. Altogether, these results suggest that putting more weight on the text similarity (versus bids) leads to nigher-expertise reviews.
Second, we investigate the "cost of randomization" to prevent fraud, measuring the effect of increasing q and thereby reducing randomness in the optimized random assignment. We consider values between q = 0.4 and g (optimal deterministic assignment). Recall the on on-policy has g = 0.5. When varying g. we find that except for a small increase in the region around q = 0.75, the average expertise for policies with y > 0.5 is very similar to that of the on-policy (Figures IA-D, right). These results suggest that using randomized instead of a deterministic policy does not lead to a significant reduction in self-reported expertise, an observation that should be contrasted with the previously documented reduction in the expected sum-siiilartty objective under randomized assignment [16]; see further analysis in Appendix I.
AAAL We perform three analyses on the AAAI data, shown in Figure I (right). First, we examine the effect of interpolating between the text-similarity scores and the subject area scores by varying Wtext E [0. again considering only deterministic policies (i.e., g = 1). The on-policy sets
8

---

TPDP
AAAI
A
Manski Bounds
Manski Bounce
Doertee
2.-
Expertee
3.5
244
2.5-
0.00
0.25
0.50
0.75
1.00
04
0.8
0.8
1.0
Weet
Aead
Model
32-
4.-
Eoertie
2.+
Experice
3.0
2.0
2.0
0.00
0.25
0.50
0.75
1.00
0.4
0.00
0.25
0.50
0.75
1.000
0.4
0.8
0.8
1.0
Maxi
Askl
C
Monotonicity Bounds
Monatonicity Bounds
Exeribee
2.8
Experice
8.5-
8.0-
2.4
2.5-
2.-
0.4
0.25
0.00
0.75
D
Lipchhtz Bounds
Lipochtz Bounds
8.2
4.0-
Dxperiss
Experèpe
8.0-
2.5-
2.0-
2.0
0.00
0.25
0.50 Whet
0.75
1.00
0.4
0.6
0.8
1.0
0.26
0.00
0.76
1.00
0.4
0.6
0.8
1.0
Whnet
Adas
Kg
AUT)
Model Impuiztion, P
ord-ppobbt
dh-oxde+
dt-imn
Paince
TPOP Lipschtzz Pail
L-000
Lipachitz, Pars
Figure 1: Expertise of off off-pooicies varying Wtext and g for TPDP and Wrext, Abia and q for AAAL computed using the different estimation methods described in Section 4. The dashed blue lines indicate Manski bounds around the on-policy expertise, and the grey lines indicate Manski bounds around the off-policy expertise. The eтrоr bands (denoted p for the Manski bounds, FBIM for the monotonicity bounds, and P for the Lipschitz bounds) represent confidence intervals that asymptotically contain the true value of μæ with probability at least 95%, as described in Appendix C. We observe similar patterns using confidence as outcome, as shown Figure the Appendix. Note that to focus on the most relevant regions of the plots, the vertical axes do not start at zero.
Wtext = 0.75. Due to large numbers of positivity violations, the Manski bounds are uninformative (Figure IE, left), so we turn the other estimators. The model imputation analysis indicates that policies with Wtext 2 0.75 may have slightly higher expertise than the on-policy and indicates lower expertise in the region where Weext S 0.5 (Figure IF, left). However, the models differ somewhat in their predictions for low Wtext, indicating that the assumptions made by these models may not be reliable. The monotonicity bounds more clearly indicate low expertise compared to the on-policy when Wtext < 0.25, but are also slightly more pessimistic about the Wtext 2 0.75 region than the models (Figure IG, left). The Lipschitz bounds indicate slightly higher than on-policy expertise for Wtext 2 0.75 and potentially suggest slightly lower than on-policy expertise for Wtext S 0.25 (Figure HH, left). Overall, all methods of analysis indicate that low values of Wtoxt result worse assignments, but the effect of considerably increasing Wtext is unclear.

---

Second, we examine the effect of increasing the weight on positive bids by varying the values of Abid- Recall that Apid corresponds to the on-policy and a higher (respectively lower) value of Apid indicates greater (respectively lesser) priority given to positive bids relative to neutral/negative bids. We investigate policies that vary Abid within the range [0, 3), and again consider only deterministic policies (i.e., q = 1). The Manski bounds are again too wide to be informative (Figure 1E, middle). The models all indicate similar values of expertise for all values of Abid and are all slightly more optimistic about expertise than the Manski bounds around the on-policy (Figure IF, middle). The monotonicity and Lipschitz bounds both agree that the Abid 2 1 region has slightly higher expertise as compared to the on- on-policy (Figure IG-H, middle). Overall, our analyses provide some indication that increasing Abid may result in slightly higher levels of expertise.
Finally, we also examine the effect of varying q within the range [0.4, 1]. Recall that the on-policy sets g = 0.52. We see that the models (Figure IF, right), the monotonicity bounds (Figure IG, right), and the Lipschitz bounds (Figure IH, right) all strongly agree that the region q 2 0.6 has slightly higher expertise than the region q E [0.4, 0.6]. However, the magnitude of this change is small, indicating that the "cost of randomization" is not very significant.
Power Investigation: Purposefully Bad Policies. As many of the off-policy assignments we consider have relatively similar estimated quality, we also ran additional analyses to show that our methods can discern differences between good policies (optimized toward high reviewer-paper similarity assignments) and policies intentionally chosen to have poor quality ("optimized" toward low reviewer-paper similarity assignments). We refer the reader to Appendix for further discussion.
6 Discussion and Conclusion
In this work, we evaluate the quality of off-policy reviewer-papee assignments in peer review using data from two venues that deployed randomized reviewer assignments. We propose new techniques for partial identification that allow us to draw useful conclusions about the off-policy review quality, even in the presence of large numbers of positivity violations and missing reviews. For a more extensive treatment of the methods proposed in this work, we refer the reader to Khan et al. [45].
Limitations. One limitation of off off-policy evaluation is that our ability to make inferences inherently depends on the amount of randomness introduced on-policy. For instance, if there is a small amount of randomness, we will be able to estimate only policies that are relatively close to the on-policy unless we are willing to make some assumptions. The approaches presented in this work allow us to examine the strength of the evidence under a wide range of types and strengths of assumptions (model imputation, boundedness of the outcome, monotonicity, and Lipschitz smoothness) and to test whether these assumptions lead to converging conclusions. We emphasize that these assumptions (except for the boundedness of the outcome) are fundamentally unverifiable in our setting, given that the outcomes for unassigned pairs are unobserved. Thus, our resulting conclusions about the quality of alternative assignments cannot be directly verified without actually deploying the off-policies in question. Additionally, the measurement of review quality is difficult question for any evaluation method: our analyses use the reviewers' self-reported expertise and confidence scores as proxies for review quality, but the self-reported nature of these measures makes them subject to various biases. Our conclusions should be interpreted in light of the possibility of misalignment between self-rep reffrreporeed expertise/confidence and the true review quality. Finally, our substantive conclusions are based on analyses of data from two venues, and thus further work is needed to test their generalizability.
Future Work. In the context of peer review, the present work considers only few parameterized slices of the vast space of reviewer waperr assignment policies, while there are many other substantive questions that our methodology can be used to answer. For instance, one could evaluate assignment quality under a different method of computing similarity scores (c.g.. different NLP algorithms [46]), additional constraints on the assignment (e.g., based on seniority or geographic diversity [6]), or objective functions other than the -fiii--iimllliiiies (e.g.. various fairness- objectives [35, 36, 47, 48)). Additional thought should also be given to the trade-offs between maximizing review quality and broader considerations of reviewer welfare: while assignments based on high text similarity may yield slightly higher-quality reviews, reviewers may be more willing to review again if the assignment policy follows their bids more closely. Beyond peer review, our work is applicable to off-policy evaluation in other matching problems, including education [49, 50), advertising [21], and ride-sharing [22].
10

---

Acknowledgments and Disclosure of Funding
We thank Gautam Kamath and Rachel Cummings for allowing to conduct this study in TPDP and Melisa Bok and Celeste Martinez Gomez from OpenReview for helping us with the Open APIs. We are also grateful to Samir Khan and Tal Wagner for helpful discussions. This work was supported in part by NSF CAREER Award 2143176, NSF CAREER Award 1942124, NSF CIF 1763734, and ONR N000142212181.
References
[1] Jim McCullough. First comprehensive survey NSF applicants focuses on their concerns about proposal review. Science, Technology, & Human Values, 1989.
[2] Marko Rodriguez, Johan Bollen, and Herbert Van Sompel. Mapping the bid behavior of conference referces. Journal of Informetrics, 1(1):68-2,, 2007.
[3] Tere Thom Jakobsen and Anna Rogers. What factors should paper-reviewer assignments rely on? community perspectives on issues and ideals in conference peer-review. In Conference of the North American Chapter of the Association for Computational Linguistics, pages 4810-4823, 2022.
[4] Nihar Shah. Challenges, experiments. and computational solutions in peer review. Communications of the ACM, 65(6):66888 87, 2022.
Laurent Charlin, Richard S. Zemel, and Craig Boutilier. A framework for optimizing paper matching. In Uncertainty Artificial Intelligence, volume 11, pages 86-95, 2011.
Kevin Leyton-Brown, Mausam, Yatin Nandwani, Hedayat Zarkoob, Chris Cameron, Neil Newman, and Dinesh Raghu, Matching papers and reviewers large conferences. arXiv preprint arXiv:2202.12273, 2022.
Simon Price and Peter Flach. Computational support for academic peer review: perspective from artificial intelligence. Communications of the ACM, 60(3):70-79, 2017.
[8] Baochun Li and Y Thomas Hou. The new automated IEEE INFOCOM review assignment system. JEEE Network, 30(5):18 24, 2016.
[9] Neil D. Lawrenee. The NIPS experiment. httpp:///aaerrprrraaaallyyo///// the n-w-niss-eeeee eaas-eexpemment.. 2014. Accessed May 17, 2023.
[10] Eric Price. The NIPS experiment. http://blog wrtz. wttp://.r//////t//////////eeeapp exteemimmm.........tt........... html, 2014. Accessed May 17, 2023.
[11] Andrew Tomkins, Min Zhang, and William D. Heavlin. Reviewer bias in single- versus double-blind peer review. Proceedings the National Academy of Sciences, 114(48):1270$ 12713, 2017.
[12] Ivan Stelmakh, Charvi Rastogi, Nihar B Shah, Aarti Singh, and Hal Daumé III. A large scale randomized controlled trial on herding in peer-review discussions. Plos 18(7):+0287443 2023.
[13] Alina Beygelzimer, Yann Dauphin, Percy Liang, and Jennifer Wortman Vaughan. The NeurIPS 2021 consistency experiment. the neurips- -021- wwwaaissteecy-eeprriieettt/t 2021. Accessed May 17, 2023.
https://blog wwwurpp..cc/22//12222
[14] Mathias Lecuyer, Joshua Lackerman, Lamont Nelson, Siddhartha Sen, Amit Sharma, and Aleksandrs Slivkins. Harvesting randomness to optimize distributed systems. In ACM Workshop on Hot Topics in Networks, pages 178-184, 2017.
[15] Michael Littman. Collusion rings threaten the integrity of computer science research. Communications of the ACM, 2021.
[16] Steven Jeemen, Hanrui Zhang, Ryan Liu, Nihar B. Shah, Vincent Conitzer, and Fei Fang. Mitigating manipulation in peer review via randomized reviewer assignments. Advances in Neural Information Processing Systems, 2020.
[17] Charles F Manski. Nonparametric bounds on treatment effects. The American Economic Review, 80(2):319- 323, 1990.
[18] Noveen Sachdeva, Yi Su, and Thorsten Joachims. Off polico bandits with deficient support. In ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 965 065-975, 2020.
11

---

[19] Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. Recom- mendations treatments: Debiasing learning and evaluation. In International Conference on Machine Learning, pages 1670-1679. PMLR, 2016.
[20] Alexandre Gilotte, Clément Calauzènes, Thomas Nedelec, Alexandre Abraham, and Simon Dollé, Offline a/b testing for recommender systems. In ACM International Conference on Web Search and Data Mining, pages
2018.
[21] Léon Bottou, Jonas Peters, Joaquin Quiñonero- Denis X Charles, D Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. Counterfactual reasoning and lcarning systems: The example of computational advertising. Journal of Muchine Learning Research, 14(11), 2013.
[22] Alex Wood- Doughty and Cameron Bruggeman. The incentives platform at lyft. In ACM International Conference on Web Search and Data Mining, pages 1654-1654, 2022.
Wood-Doughty
[23] David Mimno and Andrew McCallum. Expertise modeling for matching papers with reviewers. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 500-509. ACM, 2007.
[24] Xiang Liu, Torsten Suel, and Nasir Memon. A robust model for paper reviewer assignment. In ACM Conference on Recommender Systems, pages 25-32, 2014.
[25] Marko Rodriguez and Johan Bollen. An algorithm to determine peer-reviewers, In ACM Conference on Information and Knowledge Management, pages 319-328. ACM, 2008.
[26] Hong Diep Tran, Guillaume Cabanac, and Gilles Hubert. Expert suggestion for conference program committces. In International Conference on Research Challenges in Information Science, pages 221-233 May 2017.
[27] Laurent Charlin Richard S. Zemel. The Toronto Paper Matching System: An automated paper-reviewer assignment system. In ICML Workshop on Peer Reviewing and Publishing Models, 2013.
[28] Graham Neubig, John Wieting. Arya McCarthy, Amanda Stent, Natalie Schluter, and Trevor Cohn. Acl reviewer matching code. https: http://gttttbcccoo arllorrgrevvieerrrrrrrrrrrr.aeee wwwprieeerr-pppr--m-aeee paper-matching. 2020. Accessed May 17, 2023.
[29] Nihar B. Shah, Behzad Tahibian, Krikamol Muandet, Isabelle Guyon, and Ulrike Von Luxburg. Design and analysis of the nips 2016 review process. Journal of Machine Learning Research, 2018.
[30] Cheng Long, Raymond Wong, Peng, and Liangliang Yc. On good and fair paper-reviewer assignment. In IEEE International Conference on Data Mining, pages 1145-1150, 12 2013.
[31] Judy Goldsmith and Robert H. Sloan. The Al conference paper assignment problem. AAAI Workshop. WS-07-10:53 57, 12 2007.
[32] Wenhin Tang, Jie Tang, and Chenhao Tan. Expertise matching via constraint-based optimization. In International Conference on Web Intelligence and Intelligent Agent Technology, pages 34-41, IEEE Computer Society, 2010.
[33] Peter A. Flach, Scbastian Spiegler, Bruno Golénia, Simon Price, John Guiver, Ralf Herbrich, Thore Graepel, and Mohammed J. Zaki. Novel tools to streamline the conference review process: Experiences from SIGKDD'09. SIGKDD Explorations Newsletter, 1 11()):3367, May 2010.
[34] Camillo Taylor. On the optimal assignment of conference papers to reviewers. Technical report, Department Computer and Information Science, University of Pennsylvania, 2008.
[35] Ivan Stelmakh, Nihar B. Shah, and Aart Singh. PeerReview4All:: Fair and accurate reviewer assignment in peer review. In Algorithmic Learning Theory, 2019.
[36] Ari Kobren, Barna Saha, and Andrew McCallum. Paper matching with local fairness constraints. In ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 1247-1257, 2019.
[37] Komal Dhull, Steven Jeemen, Pravesh Kothari, and Nihar B. Shah. The price of strategyproofing peer assessment, In AAAI Conference on Human Computation and Crowdsourcing. 2022.
[38] Ivan Stelmakh, Nihar B. Shah, Aarti Singh, and Hal Daumé III. A novice-reviewer experiment to address scarcity of qualified reviewers in large conferences. In AAAI Conference on Artificial Intelligence. volume 35, pages 3785-4793, 2021.
12

---

[39] Ines Arous, Jie Yang, Mourad Khayati, and Philippe Ludré-Maurour Peer grading the peer reviews: A dual-role approach for lightening the scholarly paper review process. In Web Conference 2021, pages 1916-1927, 2021.
[40] Donald Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology. 66(5):688, 1974.
[41] Daniel Horvitz and Donovan J Thompson. A generalization of sampling without replacement from finite universe. Journal of the American Statistical Association, 47(260):663 685, 1952.
[42] VP Godambe and VM Joshi. Admissibility and bayes estimation in sampling finite populations. i. The Annals of Mathematical Statistics, 1707-727777 1965.
[43] Guido W. Imbens and Charles F. Manski. Confidence intervals for partially identified parameters. Econo- metrica, 2004.
[44] Richard E Barlow and Hugh D Brunk. The isotonic regression problem and its dual. Journal of the American Statistical Association, 67(337):140-147, 1972.
[45] Samir Khan, Martin Saveski, and Johan Ugander, Off-policy evaluation beyond overlap: partial identifica- tion through smoothness. arXiv preprint arXiv:2305. 11812. 2023.
[46] Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S Weld. Specter: Document-level representation learning using citation-informed transformers. Annual Meeting of the Association for Computational Linguistics, pages 2270-2282, 2020.
[47] Justin Payan and Yair Zick. will have order! optimizing orders fair reviewer assignment. In International Joint Conference on Artificial Intelligence, 2022.
[48] Jing Wu Lian, Nicholas Mattei, Renee Noble, and Toby Walsh. The conference paper assignment problem: Using order weighted averages to assign indivisible goods. In AAAI Conference on Artificial Intelligence, volume 32, 2018.
[49] David J Deming, Justine S Hastings, Thomas J Kane, and Douglas O Staiger. School choice, school quality, and postsecondary attainment. American Economic Review, 104(3)9991-0013 2014.
[50] Joshua D Angrist, Parag A Pathak, and Christopher R Walters. Explaining charter school effectiveness. American Econumic Journal: Applied Economics, 5(4):1-27, 2013.
[51] Yichong Xu, Han Zhao Xiaofei Shi, and Nihar B Shah. On strategyproof conference peer review. In International Joint Conference Artificial Intelligence, pages 616-622, 2019.
[52] Kevin Leyton-Brown and Mausam. AAAI 2021 introduction. https: //sl iddeliiee..... com333222/998 saai-2021 itt.:iittccctoooooaccoooooooo-ee -o13------------..e.e minute 8 onwards in the video,
2021.
[53] David Roxbee Cox. Planning of experiments. Wiley, 1958.
[54] Johan Ugander, Brian Karrer, Lars Backstrom, and Jon Kleinberg. Graph cluster randomization: Network exposure to multiple universes. In ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 329-337, 2013.
[55] Martin Saveski, Jean Pouget-Abade Guillaume Saint-Jacques, Weitao Duan, Souvik Ghosh, Ya Xu, and Edoardo M. Airoldi. Detecting network effects: Randomizing over randomized experiments. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1027-1035, 2017.
[56] Susan Athey, Dean Eckles, and Guido W. Imhens. Exact pavuue for network interference. Journal of the American Statistical Association, 1 113(521) (3522)):300-200 2018
13

---

Counterfactual Evaluation of Peer-Review Assignment Policies Supplemental Material
Martin Saveski, Steven Jecmen, Nihar B. Shah, Johan Ugander
A Linear Programs for Peer-Review Assignment
Deterministic Assignment. Let Z E {0, {0,1)1RRX(P) be an assignment matrix where Zr.p denotes whether reviewer R. assigned to paper p P. Given matrix of similarity scores S E R (R/R) x/P| standard objective is to find an assignment of papers to reviewers that maximizes the sum of similarities of the assigned pairs, subject to constraints that each paper is assigned to an appropriate number of reviewers each reviewer is assigned conflicts of interest are respected [3, 27, 30-34]. Denoting the set of conflict-of -f-ittrretttttttt pairs by C R P, this optimization problem be formulated as the following linear program:
more than a maximum number of papers k. and
max 2 PEP
Σ
Srp
PER
s.t.
Σ
Znp
=
Vp P P
PER
Σ Znd S k PEP
VE
Z = 0
v(r.p) EC
Vr E R.PEP By total unimodularity conditions, this problem has an optimal solution where Zr. {0, 11, Vr R,p E P.
0 Zr.p 1
Although the above strategy is the primary method used for paper assignments in large-scale peer review, other variants of this method have been proposed and used the literature. These algorithms consider various properties in addition to the total similarity, such as fairesss [35, 36], strategyproof- ness [37, 51], envy-freeness [47] and diversity [52]. We focus on the sum-of-similarities objective here, but our off-policy evaluation framework is agnostic to the specific objective function.
Randomized Assignment. As approach strategyproofness, Jecmen al. [16] introduce the idea of using randomization to prevent colluding reviewers and authors from being able to guarantee their assignments. Specifically, the algorithm computes a randomized paper assignment, where the marginal probability P(Zr.p) of assigning any reviewer to any paper is at most parameter q E [0, 1], chosen priori by the program chairs. These marginal probabilities are determined by the following linear program, which maximizes the expected similarity of the assignment:
Σ P(Zr.p)Sr..)
max P(Z PEE
rER
s.t.
Σ P(Z
PpE
PER
Σ P(Zr.)) S
VERR R
PEP
P(Zr,)) =0
V(rp)) EC Vr WER,EE
0 P(Zr.) 5
reviewer- assignment is then sampled using a randomized procedure that iteratively redis- tributes the probability mass placed on each reviewer-paper pair until all probabilities are either zero or one. This procedure ensures only that the desired marginal assignment probabilities are satisfied, providing no guarantees on the joint distributions of assigned pairs.
B "No Interference" Assumption
Our estimators assume that there is no interference between the units, i.e., that the treatment of one unit does not affect the outcomes for the other units. In the causal inference literature, this assumption
14

---

is referred to as the Stable Unit Treatment Value Assumption (SUTVA) [53]. In the context of peer review, SUTVA implies that: (i) The quality Yr of the review by reviewer reviewing paper p does not depend on what other reviewers are assigned to paper p. and (ii) the quality also does not depend on the other papers that reviewer r' was assigned review. The first assumption is quite realistic as in most peer review systems the reviewers cannot see other reviews until they submit their own. The second assumption is important to understand, as there could be "batch effects": a reviewer may feel more or less confident about their assessment (if measuring quality by confidence) depending on what other papers they were assigned to review. We do not test for batch effects or other violations of SUTVA in this work, which typically require either strong modeling assumptions or complex experimental designs [54-56] specifically tailored for testing SUTVA, but consider important future work.
C Variance Computation and Confidence Interval Construction
In this section, we discuss how we construct confidence intervals around μæ under Manski, mono- tonicity, and Lipschitz assumptions. To construct confidence intervals, we estimate the variance of n (Y (mimpute) as follows:
Var [ūB (Y Impude
Σ
Cov|z,, ,,
N
(5.))
Yi Impice
if E I+ if E A UI- if E IAbs.
where Y
The covariance terms (taken over Z PA) are not known exactly, owing to the fact that the procedure by Jecmen et al. [16] only constrains the marginal probabilities of individual reviewer- eviewwe-papprr pairs, but pairs of pairs can non-trivially correlated. the absence of a closed-form expression, we use Monte Carlo methods to tightly estimate these covariances. In both our analyses of the TPDP and AAAI datasets, we sampled million assignments and computed the empirical covariance. We ran an additional analysis to investigate the variability of our variance estimates. We took bootstrap sample of 100,000 assignments (from the set of all I million assignments we sampled) and computed the variance based only the (smaller) bootstrap sample. We repeated this procedure 1,000 times and computed the variance of our variance estimates. We found that the variance of our variance estimates is very small (less than 10-9) even when we use 10 times fewer sampled assignments, suggesting that we have sampled enough assignments to accurately estimate the variance.
When employing Manski bounds, we can adapt well-established inference procedure to construct 95% confidence intervals of a partial identification region that asymptotically contain the true value of uB with probability at least 95%. Following Imbens and Manski [43], we construct the interval:
p E TiB (Umin) - Lo.. V Var(P M/N, Ti (Vmmax) + 2/c.n Varl (
where the z-score analog Z',, (a 0.95), is set by their procedure such that the interval asymptot- ically has at least 95% coverage under plausible regularity conditions; for further details, see the discussion below. When employing the monotonicity assumption, we construct the interval:
PBJM
BB Many Z'on
(V
Varlii (Ý Mon)))))) ]]N,, PB (Y-Mon) + +a'an Varl (V YyMon MMMMa)///// )//N min
and when employing the Lipschitz assumption, we construct the interval:
CI E (Po(Fiin)) - z'a.
(Vrh. + z'a.n V Varliin (V ( //N
These intervals converge uniformly to the specified a-level coverage under a set of regularity assumptions on the behavior of the estimators of the upper and lower endpoints of the interval estimate: Assumption 1 from [43], establishing the coverage result in Lemma 4 there. It is difficult to verify whether Assumption satisfied for the designs (sampling reviewer-paper matchings) and interval endpoint estimators (Manski, monotonocity, Lipschitz) in this work.
15

---

A different set of assumptions, most significantly that the fraction of missing data is known before assignment, support a different method for computing confidence intervals with the coverage result in Lemma 3 from [43], obviating the need for Assumption 1. In our setting, small amounts of attrition (relative to the number policy-induced positivity violations) mean that the fraction of data that is missing is not exactly known before assignment, but almost. In practice, we find that the Imbens-Mannn interval estimates from their Lemma 3 (assuming known fraction of missing data) and Lemma 4 (assuming Assumption 1) are nearly identical for all three of the Manksi-, monotonicity,. and Lipschitz-based estimates, suggesting the coverage is well-behaved. A detailed theoretical analysis of whether the estimators obey the regularity conditions of Assumption is beyond the scope of this work.
D Linear Programs for Partial Identification Based on Lipschitz Smoothness
In this section, provide more detailed description of the linear programs used compute our partial identification estimates assuming Lipschitz smoothness on the correspondence between covariates and outcomes. Intuitively, the Lipschitz smoothness assumption captures the idea that expect two reviewer-paper pairs that are very similar in covariate space to have similar expertise. For covariate vectors X, and X,, define d(X,, Xy) as some notion of distance between the covariates. Then, the Lipschitz assumption states that there exists constant L such that Y - YjJ S Ld(X;, for all and j in R P.
As in our formulation for partial identification based on monotonicity (Section 4, LP (2)), we introduce surroaate values Yú and implement two-level optimization problem to address Lipschitz violations within the observed outcomes, i.e., two observed pairs are very close in covariate space but have different outcomes. We also define the universe of relevant pairs 2 O U Att U U Abs U I and very large constant V.
This results in the following pair of optimization problems:
(Ý min
Phaz)= arggmin V/ Yaie
Σ 1 Yil +
Σ
Y.W.+ +
Σ Y.Pa(Z)) SEI
150
IEIALLLAAD
I Y,l Id(X,, Xy) v(i,)) Eu,
Ymin S Y, S ymax
Vi eu...
The sign of the second objective term depends on whether a lower (negative) or upper (positive) bound being computed. The last of constraints are the same constraints used construct the Manski bounds described in Section 4, which here are combined with the Lipschitz assumption to jointly constrain the possible outcomes. In the limit, as + oc, the Lipschitz constraints become vacuous and we recover the Manski bounds. This problem can be reformulated and solved as linear program using standard techniques.
E
Model Implementation
To impute the outcomes of the unobserved reviewer-paper papper pairs, we train classification, ordinal regression, and collaborative filtering models. Classification models are suitable since the reviewers select their expertise and confidence scores from a set of pre-specified choices. Ordinal regression models additionally model the fact that the scores have natural ordering. Collaborative filtering models, contrast to the classification and ordinal regression models, do not rely on covariates and instead model the structure of the observed entries in the reviewer-paper paper outcome matrix, which is akin to user-item rating matrices found recommender systems.
In the classification and regression models, we use the covariates X; for each reviewer paper pair as input features. In our analysis, we consider the two/three component scores used to compute the similarities: for TPDP, X; = (T,, B.): for AAAI, X; (Ti, K,, B;). These are the primary components used by conference organizers to compute similarities, so we expect them be usefully correlated with match quality. Although perform analysis with this choice of covariates, one
16

---

TPDP
AAAI
cif-logiticc
cif-ridge
Expertice
ord-ogii-
ord probit-
c
cf-knn
cif-logistic
cif-ridge
Confidence
ord-oggit-
ord probioo
cf-svd++
cf-knn
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
Test Mean Absolute Error
Figure 2: Test performance of the imputation models described in Section 4 (Model Imputation), averaged across 10 random train/test splits of all observed reviewer-paper pairs. The error bars show 95% confidence intervals.
could also include various other features of each reviewer-paper pair, e.g., some encoding of reviewer and paper subject areas, reviewer seniority, etc.
To evaluate the performance of the models, we randomly split the observed reviewer-paper pairs into train (75%) and test (25%) sets, fit the models the train set, and measure the mean absolute error (MAE) of the predictions on the test set. To get more robust estimates of the performance, we repeat this process 10 times. In the training phase, we use 10-fold cross-validation tune the hyperparameters, using MAE as selection criterion, and retrain the model on the full training set with the best hyperparameters, We also consider two preprocessing decisions: (a) whether to encode the bids one-hot categorical variables continuous variables with the values described in Section 5, and (b) whether standardize the features. both cases, we used the settings that, overall, worked best (at prediction) for each model. We tested several models from each model category. To simplify the exposition, we only report the results of the two perf-performing models in each category. The code repository referenced in Section 1 contains the implementation of all models, including the sets of hyperparameters considered for each model.
Figure 2 shows the test MAE across the 10 random train/test splits (means and 95% Cls) using expertise and confidence outcomes for both TPDP and AAAI We note that all models perform significantly better than baseline that predicts the mean outcome in the train set. For TPDP, we find that all models perform similarly, except for cf-svd++, which performs slightly better than the other models, both for expertise and confidence. For AAAI, all classification and regression models perform similarly, but the collaborative filtering models perform slightly worse. This difference in performance is perhaps due to the fact that we consider a larger set of covariates for AAAI than TPDP, likely making the classification and ordinal regression models more predictive.
Finally, to estimate PB. we train each model on the set of all observed reviewer- papppr pairs, predict the outcomes for all unobserved pairs, and impute the predicted outcomes described in Section In the training phase, we use 10-fold cross-validation to select the hyperparameters and refit the model on the full set of observed reviewer- reviewer-paper paper pairs.
F Details of AAAI Assignment
In Section 5, described simplified version of the stage of the AAAI assignment procedure that we analyze, i.e., the assignment of senior reviewers to the first round submissions. In this section, describe this stage of the AAAI paper assignment more precisely.
randomized assignment was computed between 3145 senior reviewers and 8450 first-round paper submissions, independent of all other stages of the reviewer assignment. The set of senior reviewers was determined based on reviewing experience and publication record; these criteria were external to the assignment. Each paper assigned & senior reviewer. Reviewers were assigned to at
17

---

most A: = 4 papers, with the exception of reviewers with "Machine Learning" primary area or in the "Al For Social Impact" track, who were assigned at most k papers. The probability limit was q = 0.52.
The similarities
computed from text-similarity scores T; subject-area scores K; and bids B. Either the text-similarity scores or the area scores could be missing for given reviewer-pei paper pair, due either a reviewer failing to provide the needed information or due to other errors in the computation of the scores. The text-similarity scores T; were created using text-based scores from two different sources: (i) the Toronto Paper Matching System (TPMS) [27], and (it) the ACL Reviewer Matching code [28]. The text-similarity scores T. was set equal to the TPMS score for all pairs where this score was not missing, set equal the ACL score for all other pairs where the ACL score was not missing, and marked as missing if both scores were missing. The subject-area scores were computed from reviewer and paper subject areas using the procedure described in Appendix of [6].
Next, base scores S = Wtext Ti + (1 Wrext) K; were then computed with Wtext 0.75, if both T; and K; were not missing. If either T; or K; was missing, the base score was equal to the non-missing score of the two. If both were missing, the base score was as S 0. For pairs where the bid was "willing"" or "eager" and K; = 0, the base score was set as S T,
Next, final scores were computed as S. = S using the bid values "not willing" (0.05), "not entered" (1), "in a pinch" (1+0.55)aa). "willing" (1+ 1.5)bia). "eager" (1+3Abaa): with Abid = 1. If S < 0.15 and K; was not missing, the final score was recomputed as S; = min( K: /R: 0.115.. Finally, for reviewers who did not provide their profile for use in conffict-of-interes detection, the final score was reduced by 10%.
In all of our analyses, we follow this same procedure to determine the assignment under alternative policies (varying only the parameters Wrext, Abia. and g).
G Details Regarding Assumption Suitability
In this section, provide additional details the discussion in Section 5 on the suitability of the monotonicity and Lipschitz smoothness assumptions.
Monotonicity. Monotonicity assumes that when any component of the covariates increases, the review quality should not be lower. We can test this assumption on the observed outcomes: among all pairs of reviewer- -aper pairs with both outcomes observed, 65.7% (TPDP)/28.C 0% (AAAI) have a dominance relationship (X; > Xj) and of those pairs, 79.8% (TPDP)/76.49) (AAAI) satisfy the monotonicity condition. The fraction of dominant pairs for TPDP is higher since we consider only two covariates.
Lipschitz Smoothness. For the Lipschitz assumption, a choice of distance in covariate space is required. We choose the & distance, normalized in each dimension that all component distances lie in [0, 1J. and divided by the number of dimensions. For AAAI, some reviewer-pape missing covariate; if so, we impute a distance of in that component. We then choose several potential Lipschitz constants L by analyzing the reviewer-papee -aper pairs with observed outcomes.
pairs are
First, we examine the fraction of pairs of observed reviewer-paper pairs that violate the Lipschitz condition for each value of L. Figures 3 and 4 show the CCDF of L for pairs of observations (in other words, the fraction of violating observation pairs for each value of L) with respect to expertise and confidence respectively. In our experiments in Section 5, we use values of T. corresponding to less than 10%, 5%, and 1 1% violations from these plots.
Next, we examine the distances from unobserved reviewer-paper pairs to their closest observed reviewer-paper pair. In Figure 5, we show the CCDF of these distances for unobserved reviewer-papp paper pairs within a set of "relevant" pairs. We define the set of "relevant" unobserved pairs to be all pairs not supported on-policy that have positive probability in at least one policy among all off -folciils with varying Wtext with q = for TPDP, and all off- off-policies varying W.xxt and Apid with g = 1 for AAAI.
Intuitively, the Lipschitz assumptions correspond to beliefs that the outcome does not change too much as the similarity components change. As one example, for L = 30 AAAI, when one
18

---

TPDP
AAAI
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
10-1
100
10
10
10-1
100
10
10
Figure 3: CCDF of the L= |Y, Yjl/d(X,,, Xy) values for all pairs observed points, where Ys are expertise scores. The dashed lines denote the L values corresponding to less than 10%. 5%, and 1 1% violations. For TPDP. these values 30, 50, 300, respectively; for AAAL, L 30, 40, 100.
TPDP
AAAI
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
10-1
10
10
10-1
10"
10
10a
Figure 4: CCDF of the T. -Y,/ddX,( X3)) values for all pairs of observed points, where Ys are confidence scores. The dashed lines denote the L values corresponding to less than 10%, 5%, and 1% violations. For TPDP, these values 30, 60, 400, respectively; for AAAI, 20, 30, 70.
TPDP
AAAI
1.0
1.0
0.8
0.6
0.6
0.4
0.2
0.2
0.0
10-5
10-4
10-3
10
10-1
10"
10-5
10-4
10-3
10
10-1
10a
Distance
Distance
Figure 5: CCDF of the distances between each relevant unobserved reviewer-paper pair and its closest observed reviewer-pan pair. The dashed lines show the medians: 0.0014 (TPDP) and 0.001 (AAAD).
similarity component differs by 0.1, the outcomes can differ by most 1. Effectively, the imputed outcome of each unobserved pair is restricted to be relatively close to the outcome for the closest observed pair. From the distribution in Figure we observe median distances 0.0014 (TPDP) and 0.0011 (AAAI) across the pairs violating positivity under any of these off policies. We conclude that most imputed pairs are very close to some observed pair, and even large values of T. can significantly decrease the bound width when compared the Manski bounds.
19

---

TPDP
AAAI
1.0
Fraction optimrll similarity
0.8
0.5
0.4
0.2
0.0
Figure 6: The "cost of randomization" measured by the expected total assignment similarity. The plot shows the ratio between the sum of similarities under randomized assignment (LP (3)) with (q S and the of similarities under deterministic assignment (g = 1). The dashed lines show the values of set on-policy.
H Tie-Breaking Behavior
In Section 5 (Datasets), we specify policy in terms of the parameters of LP (3) (specifically, by altering the parameters q, Weext, and Abid from the on-policy values). However, LP (3) may have unique solution, and thus each policy may not correspond to unique set of assignment probabilities. Of particular concern, the on-policy specification of LP (3) does not uniquely identify the actual on-policy assignment probabilities.
Ideally, could the same tie-breaking methodology as was used in the on-policy to pick solution in each off-policy to avoid introducing additional effects from variations in tie-breaking behavior. However, this behavior fixed arbitrary tie-breaking behaviors such that the on on-policy solution to LP (3) matches the actual on-policy assignment probabilities; then use these same behaviors for all off-policies.
specified in the venues analyze. To resolve this,
In the TPDP analyses, we perturb all similarities by small constants such that all similarity values unique. Specifically, where A = 1e -8 and IR x|P is the same for all policies. To choose E, we sampled each entry uniformly at random from [0, and checked that the solution of the perturbed on-policy LP matches the on-policy assignment probabilities, resampling until does. This value of & was then fixed for all policies.
change the objective of LP (3) to ΣiER P(Z.)((1 A)S; + AEJ)
In the AAAI analyses, the larger size of the similarity matrix meant that randomly choosing an & that recovers the on-policy solution was not feasible. Instead, we more directly choose how to perturb the
similarities in order to achieve consistency with the on-policy. We change the objective of LP (3) to EXERXP P (S; <I[PA = 0)). where R is chosen for each policy by the following procedure. For each policy, is chosen be the largest value from {10-9, 10 10-3) such that the difference in total similarity between the solution of the original and perturbed LPs no greater than tolerance of 10-5, We confirmed that using this procedure to perturb the on-policy LP recovers the on-policy assignment probabilities, as desired.
I Similarity Cost of Randomization
In [16], Jeemen al. empirically analyze the "cost of randomization" in terms of the expected total assignment similarity. i.e., the objective value of LP (3), changes. This approach is also used by conference program chairs choose an acceptable level of q practice. In Figure 6, we show this trade-off between and sum-similarity (as a ratio to the optimal deterministic sum-similarity) for both TPDP and AAAL Note that in contrast, our approach this work measure assignment quality via self-reported expertise or confidence rather than by similarity. In particular, the cost of randomization for TPDP is high terms of sum-similarity but is revealed by our analysis be mild in terms of expertise (Section 5, Results).
20

---

Table 1: Expertise of bad policies (95% confidence intervals). 50 for TPDP and L
40 for
AAAL
Policy
Manski
Monotonicity
Lipschitz
TPDP Max
[2.6115, 2.7045]
[2.6551, 2.6782]
[2.6498, 2.6744]
TPDP Min
2.6126]
[2.5521, 2.5986]
[2.5521, 2.5937]
AAAI Max
[3.4756, 3,4783] 3.5213)
[3.4764, 3. 3,4809] [3.3396, 3.3443]
AAAI Min
[3.2591, 3.3846]
[3.3394, 3.3419]
J Power Investigation:
Purposefully Bad Policies
Many of the off aff-policy assignments we consider in Section have shown have relatively similar estimated quality. A possible explanation for this tendency is that most "reasonable" optimized policies are roughly equivalent in terms of quality, since our analyses only consider adjusting parameters of the (presumably reasonable) optimized on-policy. To investigate this possibility, we analyze a policy intentionally chosen to have poor quality.
Designing a "bad" policy that can be feasibly analyzed presents challenge, as the on-policies are both optimized and thus rarely place probability on obviously bad reviewer-paper pairs. To work within this constraint, we look for bad policies where all reviewer pairs with zero on-policy probability are regarded as conflicts. We then contrast the deterministic (q 1) policy that maximizes the total similarity score with the "bad" policy that minimizes it. Since the on-policy similarities are presumably somewhat indicative of expertise, we expect the minimization policy be worse.
The results of this comparison are presented in Table 1. On both TPDP and AAAI, we see that our methods clearly identify the minimization policies worse. The differences in quality between the policies becomes clearer with the addition of Lipschitz and monotonicity assumptions to address attrition. This illustrates that our methods are able distinguish good policy (the best of the best matches) from clearly worse one (the worst of the best matches). Thus, it is likely that our primary analyses are simply exploring high-quality regions of the sriignment-pollcce space, and that peer review assignment quality is often robust to the exact values of the various parameters.
K Broader Impact
We hope that our work will have a substantial positive impact on the quality of peer review processes broadly. Our methodology provides conference program chairs and other organizers with way to evaluate the quality of alternative reviewer-paper assignments that they can easily use for post- mortem analysis. The results from such analyses can be leveraged to adjust the assignment algorithm for future iterations of the conference, ideally leading to improved review quality and improved satisfaction from both reviewers and authors.
We note two potential negative impacts that our work may have. First, our methodology requires making assumptions about the unobserved outcomes (e.g.. monotonicity, Lipschitz smoothness) in order to achieve good estimates of assignment quality. If these assumptions do not hold in practice, the resulting estimates may be misleading, potentially supporting adjustments to the assignment policy that in fact decrease review quality. However, as there currently is no principled method for evaluating the quality of alternative assignment policies, the status quo is that such assignment policy choices are made based primarily on the intuition of program chairs. Our work makes the underlying assumptions explicit so that they can be directly considered.
Second, our work considers estimates of the average review quality across all assigned reviewer-paper pairs, analogous to the common com-of-simillritiee objective used for review-paper -pape assignment (see Section 2). However, past work has also considered other assignment objectives based on the fairness of the assignment [35, 36, 47, 48]. By focusing only on the average review quality, our analysis does not consider the impact on individual reviewers or papers, potentially encouraging assignment policies that produce higher overall review quality at the cost of fairness. In this work, we focused on average review quality as it the primary metric currently used by conference organizers, but consider generalizing our approach to estimands based on other notions of fairesss a very important direction for future work.
21

---

L Results for Confidence Outcomes
Figure shows the results of our analyses using confidence as quality measure (Y), We find that the results are substantively very similar to those reported in Section 5 using expertise.
TPDP
AAAI
Manaki Bounds
Manski Bouncs
4.0-
Confidence
Confidences
3.-
85-
32
2.0
28-
1.5
1.00
0.
Agal
Model Imputation
Imputation
Confidenss
8.-
Confidence
2.5-
28
0.00
1.00
0.4
1.0
0.00
0.75
0.50
0.75
1.00
0.4
0.5
0.8
1.0
Whee
Wise
Asd
Monotonialty Bounds
Monatonicity Bounds
8.5-
Confidence
3.-
Confelence
3.0-
32
2.0-
1.5
0.25
0.75
1.00
Uppccitz Bounds
Lipechitz
4.0
Conlidence
3.6
Coniidence
2.5
3.2
2.0
28-
0.00
0.55
0.50
0.75
1.00
0.4
0.6
0.8
1.0
0.00
0.26
0.00
0.75
1.00
0.4
0.6
0.8
1.0
Whsst
Whee
Acki
Mansal m
Model Imputstion, P
dl-iootesc:
dt-ridge
crd-logi
ord--erdt
cl-am
Par
TPOP Lipachiz PaiL
Lipachitz, Pal.
Figure 7: Confidence of off off-poliices varying Weext and g for TPDP and Wtexts Abid- and g for AAAL computed using the different estimation methods described in Section 4. The dashed blue lines indicate Manski bounds around the on-policy confidence, and the grey lines indicate Manski bounds around the off-policy confidence. The error bands (denoted ni! for the Manski bounds, PBIM for the monotonicity bounds, and PEL for the Lipschitz bounds) represent confidence intervals that asymptotically contain the true value of μB with probability at least 95% described in Appendix C. Note that focus on the most relevant regions of the plots, the vertical axes not start zero.
22


Reviews:
[{"review_id": "3uM3DWWdre", "reviewer": "Reviewer_rhmn", "summary": "This work uses the randomness of a recently-implemented peer review paper assignment algorithm in order to perform off-policy evaluation of other (nearby) randomized assignment strategies. It uses multiple methods for imputing missing values and estimating the errors in the estimators of alternative policies' relative value, including fitting proxy values to the data which obey two plausible structural assumptions. The authors perform this evaluation on paper-reviewer matching data from two conferences, and identify practical and actionable recommendations for modifications to the similarity score calculation and assignment algorithm as currently implemented.", "questions": "In your discussion of review quality, you mention competing definitions of/proxies for review quality. How does your choice (self-reported expertise and confidence) compare against the other metrics (eg auto-generated and meta-reviewer-evaluated), and would you expect your results to be robust to this choice?\n\nThe randomized paper assignment scheme of [17] which you take as the logging policy generates a fractional assignment based on the q constraints and then performs dependent rounding in order to generate an assignment. Since your estimators are linear in these assignments, I presume their means are unaffected by which rounding scheme is used. But what about the variances? (What rounding scheme is used in [17]?) Can you comment on how the choice of rounding scheme might affect the uncertainties as well as your uncertainty bounds, and whether this plays a role in your analysis?\n\nMinor comments (no response expected): \n\tYou might make the definition of review quality which you work with more prominent.\n\tThe choice of T_i for surrogate quality as well as text weight was confusing (unless these were surrogate text values, though that was not my understanding).", "limitations": "It would have been great if this work had included more data for analysis, especially from a second large conference. The scope of the recommendations offered seem somewhat limited by conferences which are a part of the analysis. It would be especially interesting to see this analysis performed on multiple years of AAAI data (provided there is a consistent on-policy used) simultaneously.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper has a clear premise, purpose, and scope, and it is well executed.\n\nIt introduces structural assumptions into the this off-policy evaluation framework which are technically and theoretically interesting and apparently novel.\n\nIt provides actionable recommendations for improving the implementation of paper-reviewer matching algorithms in a major conference, and a framework for performing similar evaluations going forward.", "weaknesses": "It seems hard to say how far these recommendations generalize beyond the conferences which are included in the analysis. \n\nIt would have been nice to have slightly more explicit discussion of how the various uncertainty estimates are generated in the main body of the work.\n\nIt would have been interesting to see this methodology used to compare the current method to a wider range of others---even deterministic similarity-based approaches---which have seen prior use.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review—we very much appreciate your feedback and are pleased to hear that you found the paper well-executed, the off-policy evaluation framework technically and theoretically interesting, and the recommendations based on our analyses actionable. \n\n> In your discussion of review quality, you mention competing definitions of/proxies for review quality. How does your choice (self-reported expertise and confidence) compare against the other metrics (eg auto-generated and meta-reviewer-evaluated), and would you expect your results to be robust to this choice?\n\nWe decided to report the results of both the reviewers’ self-reported expertise (Section 5) and confidence (Appendix L) specifically because we wanted to demonstrate that our substantive results are robust to the particular choice of a review quality metric. The fact that the analyses based on both measures lead to the same substantive conclusions in both datasets gives us reason to believe that the findings would be robust to other related outcome measures as well. We also considered using the scores provided by the meta-reviewers; however, for AAAI 85% of the meta-reviewers’ scores were missing, and for the TPDP workshop these scores were not at all collected by the conference organizers. We note that similar to reviewers’ self-reports, the meta-reviewer scores and other alternative measures, e.g., the authors’ assessment of the reviews, come with some subjective biases as well. For instance, the meta-reviewers’ scores may depend on their seniority, and the authors’ reports depend on the overall rating of the paper (see Section 8.2.2 in [4]). Last, we note that our framework and the methods proposed are general and can be applied to any outcome / review quality measure available to conference organizers.\n\n[4] Nihar B. Shah. Challenges, experiments, and computational solutions in peer review. Communications of the ACM, 65(6):76–87, 2022.\n\n\n>The randomized paper assignment scheme of [17] which you take as the logging policy generates a fractional assignment based on the q constraints and then performs dependent rounding in order to generate an assignment. Since your estimators are linear in these assignments, I presume their means are unaffected by which rounding scheme is used. But what about the variances? (What rounding scheme is used in [17]?) Can you comment on how the choice of rounding scheme might affect the uncertainties as well as your uncertainty bounds, and whether this plays a role in your analysis?\n\nThat’s an excellent question that we were also concerned about while developing the analysis framework. Your observation that there is a certain “degree of freedom” in how a deterministic assignment is generated given a fractional assignment is correct. The assignment sampling (or rounding) procedure proposed by Jecmen et al. [17] only guarantees that the marginal probabilities are respected and may choose an arbitrary distribution that does so. The covariance structure of this distribution will affect the variance of our estimators. If the off-policy evaluation was premeditated, it would be an excellent idea to devise an assignment sampling procedure that would proactively minimize the variance and maximize the power of the off-policy evaluation estimates. There are two reasons we opted to use the original procedure proposed by Jecmen et al. [17]. First and foremost, it is the procedure that had already been implemented in OpenReview and other peer-review platforms, and was already used by the conference organizers to generate the pre-existing assignment data we analyzed. Thus, efforts to improve this design would not have been useful to us in this work (but it would be an interesting direction for future work). Second, in practice, we found that accounting for the positivity violations contributes significantly more to the width of the uncertainty intervals than the variance of the estimates in the overlapping regions (where some form of covariance optimization would come in). This observation motivated us to focus our efforts on mitigating the impact of the positivity violations via our proposed estimation methods based on monotonicity and Lipshchtiz continuity assumptions. \n\n[17] Steven Jecmen, Hanrui Zhang, Ryan Liu, Nihar B. Shah, Vincent Conitzer, and Fei Fang. Mitigating manipulation in peer review via randomized reviewer assignments. Advances in Neural Information Processing Systems, 2020.\n"}, {"review_id": "jJjWZgpVCq", "reviewer": "Reviewer_4Cne", "summary": "The authors consider the problem of evaluating alternative reviewer matching algorithms. They motivate this problem as the cost of running A/B tests. They propose to use off-policy evaluation by exploiting the randomness in review assignments introduced by a fraud-mitigating scheme in recent years. They tackle several main challenges, including positivity and attrition, using imputation, Manski bounds, and improved bounds under monotonicity and continuity assumptions. They apply their method to two CS venues and their findings suggest that randomization does not significantly hurt review quality and higher review quality can be obtained by paying more attention to text similarity. ", "questions": "\n- The authors consider self reported expertise and confidence as measures of review quality - I could see these as good, but also noisy and limiting proxies. What do the authors expect are the limitations of this approach?\n\n- I'm having trouble telling whether the results given in Section 5 are verified? That is, can we tell whether the proposed approaches improve estimation (beyond the fact that the authors' approaches do what they say they will do, e.g., that the assumptions result in tighter bounds)? ", "limitations": "The authors discuss a possible limitation in their conclusion. They also discuss their assumptions in-line and refer readers to Appendix G for a discussion on assumption suitability. I do think the authors would benefit from a greater discussion of limitations (particularly if the answer to the second question above is no) and their assumptions in one place. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Sections 1 and 2 are beautifully written. \n- The authors have a clear set of challenges that they seek to addresse. \n- The authors run comprehensive experiment that yield interesting takeaways. ", "weaknesses": "- I would suggest stating the assumptions clear and upfront. For example, should state Y_i is not a function of the assignment and no interference assumption. \n\n- The challenges subsection of Section 3 is a little messy compared to the rest of the writing. One way the authors could improve this is by introducing an explicit problem statement. At the moment, it's a mix of discussion, problem statement, and assumptions.\n\n- Section 4 could be improve organizationally. The authors should clearly delineate the proposed approaches & assumptions that go with each. \n\n- The experiment section is strong but could be improved in terms of (i) explanation and (ii) organization of takeaways. For example, the figure is not explained in the text, and it takes some work to understand/parse the notation.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful comments! Thank you also for your recommendations on the exposition of the results and the organization of the manuscript. We greatly appreciate them and look forward to improving the clarity of the manuscript with these recommendations in mind.\n\n\n>The authors consider self reported expertise and confidence as measures of review quality - I could see these as good, but also noisy and limiting proxies. What do the authors expect are the limitations of this approach?\n\nWe do agree that the self-reported expertise and confidence measures that we use in this work can be noisy proxies for review quality: for example, reviewer self-reported expertise may be miscalibrated with true expertise. Similar concerns motivated us to report the results of both the reviewers’ self-reported expertise (Section 5) and confidence (Appendix L). The fact that the analyses based on both measures of review quality lead to the same substantive conclusions in the two datasets gives us reason to believe that the findings would be robust to other similar outcome measures. We additionally note that our framework and methods are general, and thus can be applied to any outcome/review quality measure available to conference organizers. Such alternative review quality measures could include asking the authors (or meta-reviewers) to rate the quality of the reviews. However, these measures can also be noisy: the authors’ reports may depend on the overall rating of the paper, and the scores of the meta-reviewers may depend on their seniority. Finally, in practice, many venues do not ask meta-reviewers / ACs to score the reviews they oversee, and when they do those scores are often missing, making off-policy evaluation even more challenging. As each measure provides an evaluation of a slightly different aspect of the review process and notion of review quality, the appropriate proxy to use may depend on the goals of the conference organizers. \n\n\n> I'm having trouble telling whether the results given in Section 5 are verified? That is, can we tell whether the proposed approaches improve estimation (beyond the fact that the authors' approaches do what they say they will do, e.g., that the assumptions result in tighter bounds)?\n\nThank you for encouraging us to think further about verification—the problem setting is such that the ground-truth outcomes are unobserved and cannot be used to directly verify the estimates. Without making assumptions about the outcomes for reviewer-paper pairs violating positivity, the Manski bounds are the only applicable estimates, as they result from imputing minimal and maximal outcomes for these pairs (Section 4). Thus, each of our proposed approaches aims to improve the estimation by leveraging some assumption about these positivity-violating pairs (e.g., the monotonicity and Lipschitz continuity assumptions). \n\nSo, while these assumptions again cannot be directly verified on the unobserved outcomes, there are many ways that we indirectly verify that they are reasonable. First, we carefully assess the suitability of these assumptions on the observed outcomes (Appendices G and E). Second, we find experimentally that a diverse set of assumptions lead to converging conclusions: e.g. the estimates of a variety of parametric models tend to lie within the same region as the estimates based on relatively mild Lipschitz/Monotonicity assumptions. This provides some evidence that these estimates are robust to violations of these assumptions; in contrast, estimates that significantly differ under different assumptions should be viewed more skeptically. \n"}, {"review_id": "nHJm54zSvo", "reviewer": "Reviewer_LpsF", "summary": "This paper leverages recently proposed strategies that introduce randomness in peer-review assignment—in order to mitigate fraud—as a valuable opportunity to evaluate counterfactual assignment strategies.\nThis paper introduces novel methods for partial identification based on monotonicity and Lipschitz continuity assumptions for the mapping between reviewer-paper covariates and outcomes. The proposed methods are applied to peer-review data from two computer science venues, the authors find that placing higher weight on text similarity results in higher review quality and that introducing randomization in the reviewer-paper assignment only marginally reduces review quality.\n", "questions": "1. Why the proposed off-policy evaluation is effective?\n2. Compared with other evaluations, can the proposed evaluation method find a better assignment policy?\n", "limitations": "The authors provide the limitations in appendix K.\n1. The method requires making assumptions: monotonicity and Lipschitz continuity.\n2. The method considers estimates of the average review quality across all assigned reviewer-paper pairs, analogous to the common sum-of-similarities objective used for review-paper assignment, the analysis does not consider the impact on individual reviewers on papers.\n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper proposes off-policy evaluation as a less costly alternative that exploits existing randomness to enable the comparison of many alternative policies, compared with A/B tests.\n2. The paper is well written and organized.\n3. The experiments are sufficient and convincing for the conclusion. \n", "weaknesses": "It would be better to compare the proposed off-policy evaluation method with previous commonly used evaluation methods, and analyze the effectiveness of this method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback! We are pleased to hear that you found the paper well-written and the experiments convincing. Below we provide responses to the questions posed in the review.\n\n\n> Why the proposed off-policy evaluation is effective?\n\nThe key idea of our proposed approach to off-policy evaluation is that we can exploit already-existing randomness (and data already generated under that randomness) in deployed reviewer-paper assignment designs. This allows us to use causal inference to answer questions about potential alternative paper assignments without intervening in the deployed assignment (which could potentially result in deploying a worse assignment); such questions cannot be credibly answered through analysis of data from deterministic assignments. This work is the first to propose leveraging randomized paper assignments to perform off-policy evaluation in the peer review setting, and our proposed methods are designed to address the challenges of applying standard techniques here (e.g., large numbers of positivity violations). \n\n\n> Compared with other evaluations, can the proposed evaluation method find a better assignment policy?\n\nIn our experimental analysis, we do find assignment policies with higher quality than the deployed policies in both venues (Figure 1). In TPDP, increasing $w_{text}$ results in a better assignment; in AAAI, increasing $w_{text}$ and $\\lambda_{bid}$ both result in slightly better assignments. We additionally find that randomized policies have a very similar quality to deterministic policies, helping to inform future conference organizers who are considering randomizing their assignments for other reasons (e.g., for the purpose of mitigating fraud). Importantly, the methods we introduce in this work can be used by conference organizers to find the best policy among any class of assignment policies they may have in mind. \n\nWe address the comparison with other evaluation methods below in response to the comments in the weaknesses section.\n\n> It would be better to compare the proposed off-policy evaluation method with previous commonly used evaluation methods, and analyze the effectiveness of this method.\n\nAs this work is the first to consider off-policy evaluation in this setting, there is only a limited set of baseline methods for comparison. In the causal inference and econometrics literature, the standard approach to estimation in the presence of positivity violations is Manski bounds, which we do consider as one of our methods. Experimentally, we find that the Manski bounds are uninformative on real conference data and are generally unable to distinguish better policies from worse ones. In contrast, our proposed methods result in much sharper estimates by carefully leveraging intuitive assumptions, allowing us to identify policies that improve over the deployed policy. \n\nIn practice, the current method for choosing the best assignment policy is highly ad-hoc: conference organizers typically generate a few sample assignments under different policies and spot-check a few of the papers’ assigned reviewers (based on their own prior knowledge). Without the techniques in this work, conference organizers have no way to evaluate an alternative, non-deployed policy in terms of the review quality (e.g., as measured by self-reported expertise). Our approach instead allows for the costless evaluation of any number of alternative policies (for which there is statistical support) based on data from the deployed assignment.\n\n"}, {"review_id": "dypxw181jo", "reviewer": "Reviewer_5Zed", "summary": "In this paper, the authors study the counterfactual evaluation of peer-review assignment strategies with randomness. The authors adapt existing off-policy evaluation methods for the specific problem to handle non-positivity, missing reviews and attribution. A framework of off-policy evaluation with different imputation choices are provided. Empirical evaluations are carried out on two datasets to study (1) effect of randomness and (2) weights on different similarity components.", "questions": "Please see above section.", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper studied an important problem as online A/B experiments for evaluating peer-review assignments are very expensive and hard to carry out.\n2. The authors propose a comprehensive framework based on off-policy evaluation with different choices of imputation methods. Moreover, the authors adapt the existing method to specific cases for paper review.\n3. The authors carry out two case studies using the proposed method on two real-world peer review datasets.\n", "weaknesses": "One major concern is that there is no evaluation on the quality for the counterfactual evaluation. It would be much more convincing if some evaluation can be carried out with the help of random A/B experiments or results from existing random experiments. Different methods provide quite different bounds. It is hard to pick the proper method without a ground-truth provided by A/B experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comments! We appreciate that you recognize the importance of the problem we consider and the value of our off-policy evaluation framework. \n\n> One major concern is that there is no evaluation on the quality for the counterfactual evaluation. It would be much more convincing if some evaluation can be carried out with the help of random A/B experiments or results from existing random experiments. Different methods provide quite different bounds. It is hard to pick the proper method without a ground-truth provided by A/B experiment.\n\nUnfortunately, no existing conference has conducted such an A/B test due to numerous significant costs and challenges. Running an A/B test in a real conference for the purpose of evaluating this work either risks substantially harming the review quality (as some reviewers are assigned under a lower-quality policy) or brings substantial costs in terms of extra reviewer time (if each paper is assigned additional reviewers). Even if we did run an A/B test, if both A and B policies were stochastic we would still likely want to use the off-policy evaluation techniques we develop here to let observations under “B” inform estimates of the “A” policy (and vice versa). Another challenge in executing an A/B test, and considering it as ground truth, is due to issues of interference: we would need to split the pool of reviewers/papers into multiple groups, and each experimental arm would likely influence the results of the other leading to biased estimates. This is a common problem in market experiments and an active research area. Given these challenges, it would be far beyond the scope of this submission to deploy an A/B test in a conference, in addition to the contributions already made in this work. \n\nOur approach in this work is instead to consider the estimates of review quality obtained under a variety of different assumptions, which we verify to be reasonable on the observed outcomes (Appendix G). Where these estimates agree (as they often do), we can draw a conclusion that is likely robust to a violation of any individual assumption; when the estimates disagree, we clearly show how the conclusion depends on which assumption one finds to be most appropriate. We also note that our approach fundamentally relies on randomization in the paper assignment to perform estimation, and thus is not observational in the sense of “observational causal inference”, a context where A/B tests are a relative “gold standard” and highly desired form of validation.\n"}]

Author rebuttal:

Meta-review: The paper studies how to evaluate peer review assignments using off-policy data (i.e. from previous randomized assignments). All the reviewers agree that the paper is timely (randomized assignments are recently becoming popular at ML conferences), well executed, clearly written, and introduces assumptions and methods that are novel, effective and theoretically interesting. The authors clarified many excellent points raised by the reviewers during the discussion, and these additional details will substantially improve the exposition in a paper revision.
Decision: Accept (spotlight)