SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yixuan Even Xu $^{1*}$  Steven Jecmen $^{2}$  Zimeng Song $^{3}$  Fei Fang $^{2}$ \
 $^{1}$ Tsinghua University\
 $^{2}$ Carnegie Mellon University\
 $^{3}$ Independent Researcher\
xuyx20@mails.tsinghua.edu.cn\
{sjcmen,feif}@cs.cmu.edu\
zmsongzm@gmail.com
The assignment of papers to reviewers is a crucial part of the peer review processes of large publication venues, where organizers (e.g., conference program chairs) rely on algorithms to perform automated paper assignment. As such, a major challenge for the organizers of these processes is to specify paper assignment algorithms that find appropriate assignments with respect to various desiderata. Although the main objective when choosing a good paper assignment is to maximize the expertise of each reviewer for their assigned papers, several other considerations make introducing randomization into the paper assignment desirable: robustness to malicious behavior, the ability to evaluate alternative paper assignments, reviewer diversity, and reviewer anonymity. However, it is unclear in what way one should randomize the paper assignment in order to best satisfy all of these considerations simultaneously. In this work, we present a practical, one-size-fits-all method for randomized paper assignment intended to perform well across different motivations for randomness. We show theoretically and experimentally that our method outperforms currently deployed methods for randomized paper assignment on several intuitive randomness metrics, demonstrating that the randomized assignments produced by our method are general-purpose.
Peer review is the process in which submissions (such as scientific papers) are evaluated by expert reviewers. It is considered a critical part of the scientific process and is commonly used to determine which papers get published in journals and conferences. For concreteness, we set this work in the academic conference setting, although the approach can be generalized to other settings, such as peer review for grant proposals and peer grading in classrooms. Due to the large scale of modern conferences like NeurIPS and AAAI, conference program chairs work closely with assignment algorithms to assign papers to reviewers automatically. Among many other challenges involved in managing huge numbers of reviewers and submissions, these organizers are faced with the difficult task of balancing various considerations for the paper assignment. Human-friendly automated paper assignment algorithms are thus crucial for helping them find desirable paper assignments.
In a standard paper assignment setting, a set  $\mathcal{P}$  of  $n_p$  papers need to be assigned to a set  $\mathcal{R}$  of  $n_r$  reviewers. To ensure each paper gets enough reviewers and no reviewer is overloaded with papers, each paper  $p$  in  $\mathcal{P}$  should be assigned to  $\ell_p$  reviewers and each reviewer  $r$  in  $\mathcal{R}$  should receive no more than  $\ell_r$  papers. An assignment is represented as a binary matrix  $\mathbf{x}$  in  $\{0,1\}^{n_p\times n_r}$ , where  $x_{p,r} = 1$  indicates that paper  $p$  is assigned to reviewer  $r$ . The main objective of paper assignment is usually to maximize the predicted match quality between reviewers and papers [1]. To characterize this,
*This work was done when Xu was a visiting intern at Carnegie Mellon University.
a similarity matrix  $\mathbf{S}$  in  $\mathbb{R}_{\geq 0}^{n_p\times n_r}$  is commonly assumed [1-7]. Here,  $S_{p,r}$  represents the predicted quality of review from reviewer  $r$  for paper  $p$  and is generally computed from various sources [8]: reviewer and paper subject areas, reviewer-selected bids, and textual similarity between the paper and the reviewer's past work [1, 9-12]. Then, the quality of an assignment can be defined as the total similarity of all assigned paper reviewers pairs, i.e.,  $\mathrm{Quality}(\mathbf{x}) = \sum_{p,r}x_{p,r}S_{p,r}$ . One standard approach for computing a paper assignment is to maximize quality [1, 4-7, 13] (which we will refer to as the maximum-quality assignment). Variants of this approach have been widely used by existing conferences, such as NeurIPS, AAAI, and ICML [8].
While the deterministic maximum-quality assignment is the most common, there are strong reasons to introduce randomness into paper assignment – that is, to determine a probability distribution over feasible deterministic assignments and sample one assignment from the distribution. Specifically, randomized paper assignments are beneficial due to the following motivations:
- Motivation 1: Robustness to malicious behavior. Several computer science conferences have uncovered "collusion rings" of reviewers and authors [14, 15], in which the reviewers aim to get assigned to the authors' papers in order to give them good reviews without considering their merits. By manipulating their stated expertise and interest (e.g., in the "paper bidding" process), these reviewers can cause the assignment algorithm to believe that their match quality with the targeted papers is very high. In other cases, reviewers may target assignment to a paper with the aim of giving it an unfair negative review [16-18]. Randomization can reduce the probability that a malicious reviewer achieves assignment to a target paper.
- Motivation 2: Evaluation of alternative assignments. Accurate reviewer-paper similarity scores are fundamental for automated paper assignment algorithms. Despite this, these scores are currently computed using various different methods by different conferences [19, 20], with no obvious way to tell beforehand which method produces the best-quality reviews. However, after deploying an assignment, it is possible to examine the resulting reviews to counterfactually evaluate the review quality produced by another method of similarity computation that is not deployed. Specifically, using techniques for off-policy evaluation [21], the randomness of the deployed paper assignment can be utilized to estimate the review quality of another non-deployed assignment. The variance of the estimation depends on the overlap in assignment probability between the deployed (randomized) assignment and the non-deployed alternative assignments of interest. Such an evaluation can then inform program chairs on how similarities should be computed (or how other algorithmic choices should be made) in the future.
- Motivation 3: Reviewer diversity. As each paper is evaluated by multiple reviewers, it is often desirable to assign a set of reviewers with diverse perspectives or areas of expertise. However, since maximum-quality assignments compute only a holistic score to represent the expertise of each reviewer-paper pair, they do not consider this factor. Randomization can increase diversity by spreading out assignment probability among a larger set of high-expertise reviewers.
- Motivation 4: Reviewer anonymity. In peer review, reviewer identities are hidden from authors so that authors cannot retaliate for negative reviews. As a result, conferences are generally reluctant to release paper assignment data since authors may be able to deduce the identities of their reviewers (even if reviewer names and other information are hidden). By sufficiently randomizing the assignment, conferences can make it difficult for authors to identify any reviewer on their paper with high probability from the assignment data.
Despite the significance of randomness in paper assignment, there is very limited prior work that looks into computing randomized assignments. A notable exception is [22], which proposed an algorithm for computing randomized paper assignments: Probability Limited Randomized Assignment (PLRA). Formally, it represents a randomized assignment as a matrix  $\mathbf{x}$  in  $[0,1]^{n_p\times n_r}$ , where  $x_{p,r}$  denotes the marginal probability that paper  $p$  is assigned to reviewer  $r$ . PLRA computes a randomized assignment via the following linear program (LP), defined for a given parameter  $Q\in [0,1]$  as:
Maximize  $\operatorname{Quality}(\mathbf{x}) = \sum_{p,r} x_{p,r} S_{p,r}$
$$
\begin{array}{c} \sum_ {r} x _ {p, r} = \ell_ {p} \\ \sum_ {n} x _ {p, r} \leq \ell_ {r} \end{array}
$$
$$
0 \leq x _ {p, r} \leq Q
$$
$$
\begin{array}{l} \forall p \in \mathcal {P}, \\ \forall r \in \mathcal {R}. \end{array}
$$
$$
\forall p \in \mathcal {P}, r \in \mathcal {R},
$$
(PLRA)
where  $\mathrm{Quality}(\mathbf{x})$  for a randomized assignment  $\mathbf{x}$  is the expected total similarity of the assignment. A deterministic assignment can then be sampled, using the fact that any feasible randomized assignment  $\mathbf{x}$  can be implemented as a distribution over feasible deterministic assignments [22, 23].
PLRA is primarily concerned with the first motivation for randomization: robustness to malicious behavior. By limiting each entry of  $\mathbf{x}$  to be at most  $Q$ , PLRA guarantees that any malicious reviewer aiming to be assigned to a target paper has at most probability  $Q$  to succeed, even if the reviewer and paper are chosen adversarially. The hyperparameter  $Q$  can be adjusted to balance the loss in quality and level of randomization. PLRA has been deployed in multiple iterations of the AAAI conference [8] and is implemented at the popular conference management system OpenReview.net [24].
However, PLRA does not fully solve the problem of randomized paper assignment. In particular, PLRA is specific to one metric of randomness: the maximum assignment probability across all paper-reviewer pairs. As a result, it is not clear how well PLRA aligns with motivations for randomization other than robustness to malicious behavior. Moreover, PLRA does not distinguish between different solutions with the same quality and maximum assignment probability. This means that it often loses opportunities to add additional randomness since it neglects to consider non-maximum assignment probabilities. One way to remedy this issue is to allow different values of  $Q$  to be set for each pair  $(p,r)$ , as in the original formulation of [22]. However, this level of flexibility makes it a significant burden for program chairs to manually choose appropriate  $Q$  values, hindering the usability of the algorithm. As a result, only the single-  $Q$  version stated above has been deployed in practice.
In this work, we address the problem of randomizing paper assignment by looking for a simple and practical method of achieving general-purpose randomized paper assignments. We consider various intuitive randomness metrics that are relevant to all above motivations but not overly specific to a particular problem formulation, and aim to provide a method that performs well across these metrics. In this way, we can provide a method for randomized paper assignment that conference program chairs can easily deploy without needing to precisely specify objectives or hyperparameters.
More specifically, we make the following contributions in this work. (1). We define several metrics to measure the extent to which the randomization in a randomized paper assignment satisfies the stated motivations (Section 3). (2). We propose Perturbed Maximization (PM), a practical algorithm for randomized paper assignment that does not rely on any specific formulation of the stated motivations (Section 4). While our algorithm can be implemented using a standard convex optimization solver, we additionally propose an approximate implementation that is computationally cheaper. (3). We provide theoretical results showing that PM provably outperforms PLRA on two classes of structured similarity scores (Section 5). (4). We extensively evaluate our algorithm via experiments using two realistic datasets of similarity scores from AAMAS 2015 and ICLR 2018 (Section 6). PM simultaneously performs well on all defined randomness metrics while sacrificing a small amount of quality as compared to the optimal non-randomized assignment. Additionally, our experiments show that PM achieves good performance when hyperparameters are set based only on the desired assignment quality, ensuring that it is simple for program chairs to deploy in practice.
This work follows a recent area of research in computer science on paper assignment algorithms for peer review. Building on the standard approach of maximum-similarity assignment, algorithms have been proposed to handle various additional considerations in the paper assignment: the fairness across papers [2], the seniority of reviewers [20], review processes with multiple phases [3], strategyproofness [25, 26], and many others [8]. We note here specifically two relevant lines of work.
One motivation for our randomized assignment algorithm is to provide robustness to malicious reviewers, a problem considered by several past works [27, 28]. Although our work most closely relates to the randomized paper assignment algorithm proposed in [22], other non-randomized approaches to the problem exist. Leyton-Brown et al. [20] describe the paper assignment process used at AAAI 2021, which included several additional soft constraints intended to curb the possibility of reviewer-author collusion rings. Wu et al. [29] propose fitting a model of reviewer bidding in order to smooth out irregular bids. Boehmer et al. [30] consider computing paper assignments without short-length cycles in order to prevent quid-pro-quo agreements between reviewers.
Randomization in the paper assignment process has also been used for evaluating different assignment policies. Traditionally, conferences will sometimes run randomized controlled experiments in order to test policy changes, where the randomization is incorporated in the assignment of reviewers to different experimental conditions. For example, WSDM 2017 randomly separated reviewers into single- and double-blind conditions in order to evaluate the benefits of double-blind reviewing [31]; other notable experiments include NeurIPS 2014 [32, 33], ICML 2020 [34], and NeurIPS 2021 [35]. In contrast, recent work by Saveski et al. [21] proposes a method for evaluating alternative assignment policies by leveraging randomization in the paper assignment itself, such as the randomized paper assignments of [22]. By introducing additional randomness at a low cost, our algorithm provides an improved basis for the methods of [21] and potential future counterfactual policy evaluation methods.
3 Metrics for Randomness and Problem Statement
In Section 1, we introduced several motivations for choosing randomized paper assignments: robustness to malicious behavior, evaluation of alternative assignments, reviewer diversity, and reviewer anonymity. These indicate that assignment quality is not the only objective that should be considered when choosing a paper assignment. In this section, we propose several metrics to characterize the extent to which the randomness in a paper assignment is practically useful.
One randomness metric considered by PLRA is maximum probability, defined as Maxprob(x) =  $\max_{p,r}\{x_{p,r}\}$ . PLRA controls Maxprob in order to trade off between quality and randomness, and thus achieves the greatest possible quality for a fixed level of Maxprob. However, as the following example shows, this metric alone is not sufficient to fully characterize the randomness of a paper assignment since it ignores the structure of the assignment with non-maximum probability.
(a) The ideal assignment
Figure 1: Example showing the limitations of PLRA. There are 2 subject areas with 5 papers and 5 reviewers.  $\ell_p = \ell_r = 1$ . The similarity of paper-reviewer pair is 1 if they are in the same area. The edge weights denote the assignment probability. (a) shows the ideal assignment with optimal quality. (b) shows an assignment by PLRA with  $Q = \frac{1}{2}$ , which is less randomized than (a) in subject area A.
(b) PLRA with  $Q = \frac{1}{2}$
Fig. 1 depicts a mini-conference consisting of 2 subject areas. Subject area A contains 3 papers and 3 reviewers, whereas subject area B contains 2 of each. The similarities between reviewers and papers within the same subject area are 1 and similarities across subject areas are 0. The constraints are  $\ell_p = \ell_r = 1$ , i.e., one-to-one assignment. Intuitively, the best randomized assignment matches papers and reviewers uniformly at random within each subject area, like Fig. 1a, since this does not sacrifice any quality. However, PLRA fails to give out this ideal assignment regardless of the hyperparameter choice. Specifically,  $Q \geq 1/2$  is required for PLRA to get a maximum-quality assignment, but when  $Q \geq 1/2$ , PLRA considers Fig. 1a and Fig. 1b to be equivalent (in terms of objective value). In fact, infinitely many solutions with the same optimal quality are considered equivalent by PLRA, and current LP solvers tend to return Fig. 1b as it is a vertex solution when  $Q = 1/2$ . In essence, PLRA is leaving "free randomness on the table," with many practical implications.
Consider the problem of mitigating malicious behavior (Motivation 1), and compare the assignments in Figs. 1a and 1b. Although they appear to have the same Maxprob, when we only look at subject area A, the Maxprob of Fig. 1a is lower than that of Fig. 1b. This indicates that Fig. 1a is more robust to malicious behavior within subject area A. Therefore, Fig. 1a is more desirable.
Moreover, as introduced in Motivation 2, randomness can also be leveraged to evaluate alternative paper assignments using observations of the review quality from a deployed, randomized assignment. These techniques rely on a "positivity" assumption: paper-reviewer pairs assigned in the alternative assignment must be given non-zero probability in the deployed assignment. Thus, if we spread assignment probability more uniformly among more reviewer-paper pairs, the resulting data can be
used to evaluate more varied strategies with tighter bounds. In this sense, the assignment in Fig. 1a will allow us to better estimate the quality of different paper assignments within subject area A.
The failure of PLRA on such a simple example shows that Maxprob alone is an inadequate metric. Thus, we need other metrics to distinguish between assignments like Fig. 1a and Fig. 1b. We therefore propose, in addition to Maxprob, a set of new randomness metrics to capture the neglected low-probability structure of an assignment. Under each of these metrics, a uniform assignment is considered "more random" than any other assignment, thus distinguishing Fig. 1a from Fig. 1b.
(1) Average maximum probability:  $\mathrm{AvgMaxp}(\mathbf{x}) = \frac{1}{n_p}\sum_p\max_r\{x_{p,r}\}$ . With respect to the motivation of preventing malicious behavior, this randomness metric corresponds to the case when a target paper is randomly chosen and the reviewer targeting assignment to that paper is adversarially chosen. By minimizing average maximum probability, we will limit the success probability of manipulation in that case.
(2) Support size:  $\mathrm{Support}(\mathbf{x}) = \sum_{p,r}\mathbb{I}[x_{p,r} > 0]$ . Support size directly relates to the "positivity" assumption introduced above. As there may be many alternative paper assignments of interest, maximizing the support size effectively maximizes the quality of estimation across them.
(3) Entropy: Entropy  $(\mathbf{x}) = -\sum_{p,r}x_{p,r}\ln (x_{p,r})$ . In information theory, entropy characterizes the uncertainty of a random variable. By maximizing entropy, we maximize the uncertainty of our assignment, corresponding to the idea of maximizing randomness. Note that strictly speaking, assignment  $\mathbf{x}$  is not a probability distribution, so this definition is a generalization.
(4) L2 norm: L2Norm(x) =  $\sqrt{\sum_{p,r}x_{p,r}^2}$ . To prevent manipulation, PLRA limits the assigned probability of each pair to be at most a specified value  $Q$ . L2 norm relaxes this constraint to a soft one: a higher probability results in a larger loss. Note that a uniformly random assignment will always have the smallest L2 norm, and so minimizing L2 norm pushes the assignment towards the uniform assignment.
The combination of these metrics more comprehensively captures the impact of randomness on the motivations from Section 1. Moreover, they do so without requiring a specific problem formulation for each motivation (e.g., an assumption on the behavior of malicious reviewers, a list of the alternative assignments of interest), which are impractical or infeasible to accurately specify in practice.
Problem statement. For an input instance  $(n_p, n_r, \ell_p, \ell_r, \mathbf{S})$ , we want to find an algorithm achieving a good trade-off between quality and randomness. Specifically, let the maximum possible quality be  $M$ . For a given lower bound of assignment quality  $\eta M$  ( $\eta \in [0,1]$ ), we want the algorithm to produce an assignment  $\mathbf{x}$  with lower Maxprob( $\mathbf{x}$ ), AvgMaxp( $\mathbf{x}$ ), L2Norm( $\mathbf{x}$ ) and higher Entropy( $\mathbf{x}$ ), Support( $\mathbf{x}$ ), i.e., a good Pareto-frontier of quality and randomness.
4 Perturbed Maximization
In this section, we present our proposed algorithm for randomized paper assignment. To describe it, we first present some definitions.
Definition 4.1 (Perturbation Function). A function  $f:[0,1]\to [0,1]$  is a perturbation function if (i)  $f(0) = 0$ , (ii)  $f'$  exists, (iii)  $f$  is non-decreasing on  $[0,1]$  and (iv)  $f$  is concave on  $[0,1]$ .
Definition 4.2 (Perturbed Quality). For an assignment  $\mathbf{x}$ , its perturbed quality with respect to perturbation function  $f$  on instance  $(n_p, n_r, \ell_p, \ell_r, \mathbf{S})$  is  $\mathrm{PQuality}_f(\mathbf{x}) = \sum_{p,r} S_{p,r} \cdot f(x_{p,r})$ .
The definition of perturbed quality incorporates the intuition from the motivating example in the previous section. As function  $f$  is concave, the marginal increase of  $\mathrm{PQuality}_f(\mathbf{x})$  from increasing a specific entry  $x_{p,r}$  is diminishing as  $x_{p,r}$  grows. Consequently, the assignment of Fig. 1a will have a higher perturbed quality than that of Fig. 1b. This naturally gives our new algorithm, Perturbed Maximization (PM). For a given parameter  $Q \in [0,1]$  and a perturbation function  $f$ :
Maximize  $\underline{\mathrm{PQuality}}_f(\mathbf{x}) = \sum_{p,r}S_{p,r}\cdot f(x_{p,r})$
Subject to  $\begin{array}{l}\sum_{r}x_{p,r} = \ell_{p}\\ \sum_{p}x_{p,r}\leq \ell_{r} \end{array}$ $\forall p\in \mathcal{P},$ $\forall r\in \mathcal{R},$  (PM)
$0\leq x_{p,r}\leq Q$ $\forall p\in \mathcal{P},r\in \mathcal{R}.$
Note that PM is a class of algorithms induced by different perturbation functions. When the perturbation function  $f$  is chosen to be a linear function, PM becomes PLRA. In the main experiments of this paper, two specific perturbation functions are considered: (1). Exponential Perturbation Function:  $f(x) = 1 - e^{-\alpha x}$  where  $\alpha \in (0, +\infty)$  and (2). Quadratic Perturbation Function:  $f(x) = x - \beta x^2$  where  $\beta \in [0, 1]$ . Respectively, we will denote PM with  $f(x) = 1 - e^{-\alpha x}$  and  $f(x) = x - \beta x^2$  as PM-Exponential (PM-E) and PM-Quadratic (PM-Q) in the rest of the paper.
In Section 6, we will empirically show that under our settings of hyperparameters, PM-E and PM-Q have almost identical performances on every metric we consider, which suggests that the specific form of the perturbation function has limited impact as long as it is strictly concave. Therefore, there is likely no need to consider many different types of perturbation functions.
To analyze PM, first notice that the concaveness of function  $f$  guarantees that the optimization program is concave. Therefore, we can use standard concave optimization methods like gradient ascent or the ellipsoid method to solve the program in polynomial time. In most of the experiments of this paper, we will use Gurobi [36], a well-known commercial solver, to solve PM. While solving PM as a general concave optimization problem is conceptually convenient, doing so also incurs a high time complexity as there are  $n_p \cdot n_r$  variables. To further speed up the execution of PM, we propose Algorithm 1, a network-flow-based approximation of PM.
Algorithm 1: Network-Flow-Based Approximation of PM