SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Martin Saveski
msaveski@uw.edu
Nihar B. Shah
nihars@cs.cmu.edu
Steven Jecmen
sjecmen@cs.cmu.edu
Johan Ugander
jugander@stanford.edu
Peer review assignment algorithms aim to match research papers to suitable expert reviewers, working to maximize the quality of the resulting reviews. A key challenge in designing effective assignment policies is evaluating how changes to the assignment algorithm map to changes in review quality. In this work, we leverage recently proposed policies that introduce randomness in peer-review assignment—in order to mitigate fraud—as a valuable opportunity to evaluate counterfactual assignment policies. Specifically, we exploit how such randomized assignments provide a positive probability of observing the reviews of many assignment policies of interest. To address challenges in applying standard off-policy evaluation methods, such as violations of positivity, we introduce novel methods for partial identification based on monotonicity and Lipschitz smoothness assumptions for the mapping between reviewer-paper covariates and outcomes. We apply our methods to peer-review data from two computer science venues: the TPDP'21 workshop (95 papers and 35 reviewers) and the AAAI'22 conference (8,450 papers and 3,145 reviewers). We consider estimates of  $(i)$  the effect on review quality when changing weights in the assignment algorithm, e.g., weighting reviewers' bids vs. textual similarity (between the review's past papers and the submission), and  $(ii)$  the "cost of randomization", capturing the difference in expected quality between the perturbed and unperturbed optimal match. We find that placing higher weight on text similarity results in higher review quality and that introducing randomization in the reviewer-paper assignment only marginally reduces the review quality. Our methods for partial identification may be of independent interest, while our off-policy approach can likely find use in evaluating a broad class of algorithmic matching systems.
The assignment of papers to reviewers is one of the most important parts of the peer-reviewed publication process [1-3]. In computer science, conferences are the primary terminal venue for publications, with recent iterations of large conferences such as NeurIPS and AAAI receiving several thousand submissions [4]. As a result, these and other large conferences must rely on automated systems to decide what members of the impaneled reviewer pool will review each paper. Review matching systems typically use three sources of information: (i) bids, i.e., reviewers' self-reported preferences to review the papers; (ii) text similarity between the paper and the reviewer's publications; and (iii) reviewer- and author-selected subject areas. Given a prescribed way to combine these
signals into a single score, an optimization procedure then proposes a reviewer-paper assignment that maximizes the sum of the scores of the assigned pairs [5].
The design of peer-review systems has received considerable research attention [4, 6-8]. Popular peer-review platforms such as OpenReview and Microsoft CMT offer many features that conference organizers can use to assign reviewers. However, it has been persistently challenging to evaluate how changes to peer-review assignment algorithms affect review quality. An implicit assumption underlying such approaches is that review quality is an increasing function of bid enthusiasm, text similarity, and subject area match, but how to combine these signals into a score is approached via heuristics. Researchers typically observe only the reviews actually assigned by the algorithm and have no way of measuring the quality of reviews under an assignment generated by an alternative algorithm.
One approach to comparing different assignment policies is running A/B tests. Several conferences (NeurIPS'14 [9, 10], WSDM'17 [11], ICML'20 [12], and NeurIPS'21 [13]) have run A/B tests to evaluate various aspects of their review process, such as differences between single- vs. double-blind review. However, such experiments are extremely costly in the peer review context, with the NeurIPS experiments requiring a significant number of additional reviews, overloading already strained peer review systems. Moreover, A/B tests typically compare only a handful of design decisions, while assignment algorithms typically require making many such decisions (Section 2).
In this work, we propose off-policy evaluation as a less costly alternative that exploits existing randomness to enable the comparison of many alternative policies. Our proposed technique "harvests" [14] the randomness introduced in peer-review assignments generated by recently-adopted techniques that counteract fraud in peer review. In recent years, in-depth investigations have uncovered evidence of rings of colluding reviewers in a few computer science conferences [15]. These reviewers conspire to manipulate the paper assignment in order to give positive reviews to the papers of co-conspirators. To mitigate this kind of collusion, conference organizers have adopted various techniques, including a recently introduced randomized assignment algorithm [16]. This algorithm limits the maximum probability (a parameter set by the organizers) of any reviewer getting assigned any particular paper. This randomization thus limits the expected rewards of reviewer collusion at the cost of some reduction in the expected sum-of-similarities objective, and has been implemented in OpenReview since 2021 and used by several conferences, including AAAI'22 and AAAI'23.
The key insight of the present work is that under this randomized assignment policy, a range of reviewer-paper pairs other than the exactly optimal assignment become probable to observe. We can then adapt the tools of off-policy evaluation and importance sampling to evaluate the quality of many alternative policies. A major challenge, however, is that off-policy evaluation assumes overlap between the on-policy and the off-policy, i.e., that each reviewer-paper assignment that has a positive probability under the off-policy also had a positive probability under the on-policy. In practice, positivity violations are inevitable even when the maximum probability of assigning any reviewer-paper pair is low enough to induce significant randomization, especially as we are interested in evaluating a wide range of design choices of the assignment policy. To address this challenge, we build on existing literature for partial identification and propose methods that bound the off-policy estimates while making weak assumptions on how positivity violations arise.
More specifically, we propose two approaches for analysis that rely on different assumptions on the mapping between the covariates (e.g., bid, text similarity, subject area match) and the outcome (e.g., review quality) of the reviewer-paper pairs. First, we assume monotonicity in the covariates-outcome mapping. Understood intuitively, this assumption states that if reviewer-paper pair  $i$  has higher or equal bid, text similarity, and subject area match than a reviewer-paper pair  $j$ , then we assume that the quality of the review for pair  $i$  is higher or equal to the review for pair  $j$ . Alternatively, we assume Lipschitz smoothness in the covariate-outcome mapping. Intuitively, this assumption captures the idea that two reviewer-paper pairs that have similar bids, text similarity, and subject area match, should result in a similar review quality. We find that this Lipschitz assumption naturally generalizes so-called Manski bounds [17], the partial identification strategy that assumes only bounded outcomes.
We apply our methods to data collected by two computer science venues that used the recently-introduced randomized assignment policy: the 2021 Workshop on Theory and Practice of Differential Privacy (TPDP) with 95 papers and 35 reviewers, and the 2022 AAAI Conference on Advancement in Artificial Intelligence (AAAI) with 8,450 papers and 3,145 reviewers. We evaluate two design choices:  $(i)$  how varying the weights of the bids vs. text similarity vs. subject area match (latter available only in AAAI) affects the overall quality of the reviews, and  $(ii)$  the "cost of randomization",
i.e., how much the review quality decreased as a result of introducing randomness in the assignment. As our measure of assignment quality, we consider the expertise and confidence reported by the reviewers for their assigned papers. We find that our proposed methods for partial identification assuming monotonicity and Lipschitz smoothness significantly reduce the bounds of the estimated review quality off-policy, leading to more informative results. Substantively, we find that placing a larger weight on text similarity results in higher review quality, and that introducing randomization in the assignment leads to a very small reduction in review quality.
Beyond our contributions to the design and study of peer review systems, the methods proposed in this work should also apply to other matching systems such as recommendation systems [18-20], advertising [21], and ride-sharing assignment systems [22]. Further, our contributions to off-policy evaluation under partial identification should be of independent interest.
Our code is available at: https://github.com/msaveski/counterfactual-peer-review.
We start by reviewing the fundamentals of peer-review assignment algorithms.
Reviewer-Paper Similarity. Consider a peer review scenario with a set of reviewers  $\mathcal{R}$  and a set of papers  $\mathcal{P}$ . Standard assignment algorithms for large-scale peer review rely on "similarity scores" for every reviewer-paper pair  $i = (r,p)\in \mathcal{R}\times \mathcal{P}$ , representing the assumed quality of review by that reviewer for that paper. These scores  $S_{i}$ , typically non-negative real values, are commonly computed from a combination of up to three sources of information: (1) text-similarity scores  $T_{i}$  between each paper and reviewer's past work, using various techniques [23-28]; (2) overlap  $K_{i}$  between the self-nominated subject areas selected by each reviewer and each paper's authors; and (3) reviewer-provided "bids"  $B_{i}$  on each paper. Without any principled methodology for evaluating the choice of similarity score, conference organizers manually select a parametric functional form and choose parameter values by spot-checking a few reviewer-paper assignments. For example, a simple similarity function is a convex combination of the component scores:  $S_{i} = w_{\mathrm{text}}T_{i} + (1 - w_{\mathrm{text}})B_{i}$ . Conferences have also used more complex non-linear functions: NeurIPS'16 [29] used the functional form  $S_{i} = (0.5T_{i} + 0.5K_{i})2^{B_{i}}$ , while AAAI'21 [6] used  $S_{i} = (0.5T_{i} + 0.5K_{i})^{1 / B_{i}}$ . The range of possible functional forms results in a wide design space, which we explore in this work.
Deterministic Assignment. Let  $Z \in \{0,1\}^{|\mathcal{R}| \times |\mathcal{P}|}$  be an assignment matrix where  $Z_{i}$  denotes whether the reviewer-paper pair  $i$  was assigned or not. Given a matrix of reviewer-paper similarity scores  $S \in \mathbb{R}_{\geq 0}^{|\mathcal{R}| \times |\mathcal{P}|}$ , a standard objective is to find an assignment of reviewers to papers that maximizes the sum of similarities of the assigned pairs, subject to constraints that each paper is assigned to an appropriate number of reviewers, each reviewer is assigned no more than a maximum number of papers, and conflicts of interest are respected [5, 27, 30-34]. This optimization problem can be formulated as a linear program. We provide a detailed formulation in Appendix A. While other objective functions have been proposed [35-37], here we focus on the sum-of-similarities objective.
Randomized Assignment. As one approach to strategyproofness, Jecmen et al. [16] introduce the idea of using randomization to prevent colluding reviewers and authors from being able to guarantee their assignments. Specifically, the program chairs first choose a parameter  $q \in [0,1]$ . Then, the algorithm computes a randomized paper assignment, where the marginal probability  $P(Z_{i} = 1)$  of assigning any reviewer-paper pair  $i$  is at most  $q$ . These marginal probabilities are determined by a linear program, which maximizes the expected similarity of the assignment subject to the probability limit  $q$  (detailed formulation in Appendix A). A reviewer-paper assignment is then sampled using a randomized procedure that iteratively redistributes the probability mass placed on each reviewer-paper pair until all probabilities are either zero or one.
Review Quality. The above assignments are chosen based on maximizing the (expected) similarities of assigned reviewer-paper pairs, but those similarities may not be accurate proxies for the quality of review that the reviewer can provide for that paper. In practice, automated similarity-based assignments result in numerous complaints of low-expertise paper assignments from both authors and reviewers [3]. Meanwhile, self-reported assessments of reviewer-paper assignment quality can be collected from the reviewers themselves after the review. Conferences often ask reviewers to score their expertise in the paper's topic and/or confidence in their review [6, 29, 38]. Other indicators of review quality can also be considered; e.g., some conferences ask "meta-reviewers" or other reviewers
to evaluate the quality of written reviews directly [38, 39]. In this work, we consider self-reported expertise and confidence as our measures of review quality.
3 Off-Policy Evaluation
One attractive property of the randomized assignment described above is that while only one reviewer-paper assignment is sampled and deployed, many other assignments could have been sampled, and those assignments could equally well have been generated by some alternative assignment policy. The positive probability of other assignments allows us to investigate whether alternative assignment policies might have resulted in higher-quality reviews.
Let  $A$  be a randomized assignment policy with a probability density  $P_A$ , where  $\sum_{Z \in \{0,1\}^{|\mathcal{R}| \times |\mathcal{P}|}} P_A(Z) = 1$ ;  $P_A(Z) \geq 0$ ,  $\forall Z$ ; and  $P_A(Z) > 0$  only for feasible assignments  $Z$ . Let  $B$  be another policy with density  $P_B$ , defined similarly. We denote by  $P_A(Z_i)$  and  $P_B(Z_i)$  the marginal probabilities of assigning reviewer-paper pair  $i$  under  $A$  and  $B$  respectively. Finally, let  $Y_i \in \mathbb{R}$ , where  $i = (r,p) \in \mathcal{R} \times \mathcal{P}$ , be the measure of the quality of reviewer  $r$ 's review of paper  $p$ .
We follow the potential outcomes framework for causal inference [40]. Throughout this work, we will let  $A$  be the on-policy or the logging policy, i.e., the policy that the review data was collected under, while  $B$  will denote one of several alternative policies of interest. In Section 5, we will describe the specific alternative policies we consider in this work. Define  $N = \sum_{i\in \mathcal{R}\times \mathcal{P}}Z_i$  as the total number of reviews, fixed across policies and set ahead of time. We are interested in the following estimands:
$$
\mu_ {A} = \mathbb {E} _ {Z \sim P _ {A}} \left[ \frac {1}{N} \sum_ {i \in \mathcal {R} \times \mathcal {P}} Y _ {i} Z _ {i} \right], \quad \mu_ {B} = \mathbb {E} _ {Z \sim P _ {B}} \left[ \frac {1}{N} \sum_ {i \in \mathcal {R} \times \mathcal {P}} Y _ {i} Z _ {i} \right],
$$
where  $\mu_{A}$  and  $\mu_{B}$  are the expected review quality under policy  $A$  and  $B$ , respectively.
In practice, we do not have access to all  $Y_{i}$ , but only those that were assigned. Let  $Z^{A}\in \{0,1\}^{\lvert\mathcal{R}\rvert\times\lvert\mathcal{P}\rvert}$  be the assignment sampled under the on-policy A, drawn from  $P_{A}$ . We define the following Horvitz-Thompson estimators of the means:
$$
\widehat {\mu} _ {A} = \frac {1}{N} \sum_ {i \in \mathcal {R} \times \mathcal {P}} Y _ {i} Z _ {i} ^ {A}, \quad \widehat {\mu} _ {B} = \frac {1}{N} \sum_ {i \in \mathcal {R} \times \mathcal {P}} Y _ {i} Z _ {i} ^ {A} W _ {i}, \quad \text {w h e r e} W _ {i} = \frac {P _ {B} \left(Z _ {i}\right)}{P _ {A} \left(Z _ {i}\right)} \forall i \in \mathcal {R} \times \mathcal {P}. \tag {1}
$$
For now, suppose that  $B$  has positive probability only where  $A$  is positive (also known as satisfying "positivity"):  $P_A(Z_i) > 0$  for all  $i \in \mathcal{R} \times \mathcal{P}$  where  $P_B(Z_i) > 0$ . Then, all weights  $W_i$  where  $P_B(Z_i) > 0$  are bounded. As we will see, many policies of interest  $B$  go beyond the support of  $A$ .
Under the positivity assumption,  $\widehat{\mu}_A$  and  $\widehat{\mu}_B$  are unbiased estimators of  $\mu_A$  and  $\mu_B$  respectively [41]. Moreover, the Horvitz-Thompson estimator is admissible in the class of all unbiased estimators [42]. Note that  $\widehat{\mu}_A$  is the empirical mean of the observed assignment sampled on-policy, and  $\widehat{\mu}_B$  is a weighted mean of the observed assignment based on inverse probability weighting. These estimators also rely on a standard causal inference assumption of "no interference"; i.e., that the outcomes  $Y_{i}$  do not depend on the assignments  $Z_j^A$  for any other reviewer-paper pair  $j\neq i$ . In Appendix B, we discuss the implications of this assumption in the peer review context.
Challenges. In off-policy evaluation, we are interested in evaluating a policy  $B$  based on data collected under policy  $A$ . However, our ability to do so is typically limited to policies where the assignments that would be made under  $B$  are possible under  $A$ . In practice, many interesting policies step outside of the support of  $A$ . Outcomes for reviewer-paper pairs outside the support of  $A$  but with positive probability under  $B$  ("positivity violations") cannot be estimated and must either be imputed by some model or have their contribution to the average outcome  $(\mu_B)$  bounded.
In addition to positivity violations, we identify three other mechanisms through which missing data with potential confounding may arise in the peer review context: absent reviewers, selective attrition, and manual reassignments. For absent reviewers, i.e., reviewers who have not submitted any reviews, we do not have a reason to believe that the reviews are missing due to the quality of the reviewer-paper assignment. Hence, we assume that their reviews are missing at random, and impute them with the weighted mean outcome of the observed reviews. For selective attrition, i.e., when some but not all reviews are completed, we instead employ conservative bounding techniques as for policy-based positivity violations. Finally, reviews might be missing due to manual reassignments by the program
chairs, after the assignment has been sampled. As a result, the originally assigned reviews will be missing and new reviews will be added. In such cases, we treat removed assignments as attrition (i.e., bounding their contribution) and ignore the newly introduced assignments as they did not arise from any determinable process. Concretely, we partition the reviewer-paper pairs into the following (mutually exclusive and exhaustive) sets:
$\triangleright \mathcal{I}^{-}$  : positivity violations,  $\{i = (r,p)\in \mathcal{R}\times \mathcal{P}:P_A(Z_i) = 0\land P_B(Z_i) > 0\}$
$\triangleright \mathcal{I}^{Abs}$ : missing reviews where the reviewer was absent (submitted no reviews),
$\triangleright \mathcal{I}^{Att}$  : remaining missing reviews, and
$\triangleright \mathcal{I}^{+}$ : remaining pairs without positivity violations or missing reviews,  $(\mathcal{R}\times \mathcal{P})\backslash (\mathcal{I}^{Att}\cup \mathcal{I}^{Abs}\cup \mathcal{I}^{-})$ . In the next section, we present methods for imputing or bounding the contribution of  $\mathcal{I}^{-}$  to the estimate of  $\widehat{\mu}_B$ , and  $\mathcal{I}^{Abs}$  and  $\mathcal{I}^{Att}$  to the estimates of  $\widehat{\mu}_A$  and  $\widehat{\mu}_B$ .
4 Imputation and Partial Identification
In the previous section, we defined three sets of reviewer-paper pairs  $i$  for which outcomes  $Y_{i}$  must be imputed rather than estimated:  $\mathcal{I}^{-}$ ,  $\mathcal{I}^{Abs}$ ,  $\mathcal{I}^{Att}$ . In this section, we describe varied methods for imputing these outcomes that rely on different strengths of assumptions, including methods that output point estimates and methods that output lower and upper bounds of  $\hat{\mu}_B$ . In Section 5, we apply these methods to peer-review data from two computer science venues.
For missing reviews where the reviewer is absent  $(\mathcal{I}^{Abs})$ , we assume that the reviewer did not participate in the review process for reasons unrelated to the assignment quality (e.g., too busy). Specifically, we assume that the reviewers are missing at random and thus impute the mean outcome among  $\mathcal{I}^+$ , the pairs with no positivity violations or missing reviews:  $\overline{Y} = (\sum_{i\in \mathcal{I}^{+}}Y_{i}Z_{i}^{A}W_{i}) / (\sum_{i\in \mathcal{I}^{+}}Z_{i}^{A}W_{i})$
In contrast, for positivity violations  $(\mathcal{I}^{-})$  and the remaining missing reviews  $(\mathcal{I}^{Att})$ , we allow for the possibility that these reviewer-paper pairs being unobserved is correlated with their unobserved outcome. Thus, we consider imputing arbitrary values for  $i$  in these subsets, which we denote by  $Y_{i}^{Impute}$  and place into a matrix  $Y^{Impute} \in \mathbb{R}^{|\mathcal{R}| \times |\mathcal{P}|}$ , leaving entries for  $i \notin \mathcal{I}^{-} \cup \mathcal{I}^{Att}$  undefined. This strategy corresponds to setting  $Y_{i} = Y_{i}^{Impute}$  for  $i \in \mathcal{I}^{-} \cup \mathcal{I}^{Att}$  in estimator (1). To obtain bounds, we impute both the assumed minimal and maximal values of  $Y_{i}^{Impute}$ .
These modifications result in a Horvitz-Thompson off-policy estimator with imputation. To denote this, we redefine  $\widehat{\mu}_B$  to be a function  $\widehat{\mu}_B: \mathbb{R}^{|\mathcal{R}| \times |\mathcal{P}|} \to \mathbb{R}$ , where  $\widehat{\mu}_B(Y^{Impute})$  denotes the estimator resulting from imputing entries from a particular choice of  $Y^{Impute}$ :
$$
\widehat {\mu} _ {B} (Y ^ {I m p u t e}) = \frac {1}{N} \left(\sum_ {i \in \mathcal {I} ^ {+}} Y _ {i} Z _ {i} ^ {A} W _ {i} + \sum_ {i \in \mathcal {I} ^ {A t t}} Y _ {i} ^ {I m p u t e} Z _ {i} ^ {A} W _ {i} + \sum_ {i \in \mathcal {I} ^ {A b s}} \overline {{Y}} Z _ {i} ^ {A} W _ {i} + \sum_ {i \in \mathcal {I} ^ {-}} Y _ {i} ^ {I m p u t e} P _ {B} (Z _ {i})\right).
$$
The estimator computes the weighted mean of the observed  $(Y_{i})$  and imputed outcomes  $(Y_{i}^{Impute}$  and  $\overline{Y}$ ). We impute  $Y_{i}^{Impute}$  for the attrition  $(\mathcal{I}^{Att})$  and positivity violation  $(\mathcal{I}^{-})$  pairs, and  $\overline{Y}$  for the absent reviewers  $(\mathcal{I}^{Abs})$ . Note that we weight the imputed positivity violations  $(\mathcal{I}^{-})$  by  $P_B(Z_i)$  rather than  $Z_{i}W_{i}$ , since the latter is undefined. Under the assumption that the imputed outcomes are accurate,  $\widehat{\mu}_B(Y^{Impute})$  is an unbiased estimator of  $\mu_B$ . We can further construct confidence intervals using the methods described in Appendix C.
Next, we detail several methods by which we choose  $Y^{Impute}$ . These methods rely on various assumptions of different strength about the unobserved outcomes.