Title: A One-Size-Fits-All Approach to Improving Randomness in Paper Assignment
Venue: NeurIPS 2023
Accepted: yes
Keywords: peer review

randomized paper assignment

mitigating malicious behavior

convex optimization
Abstract: The assignment of papers to reviewers is a crucial part of the peer review processes of large publication venues, where organizers (e.g., conference program chairs) rely on algorithms to perform automated paper assignment. As such, a major challenge for the organizers of these processes is to specify paper assignment algorithms that find appropriate assignments with respect to various desiderata. Although the main objective when choosing a good paper assignment is to maximize the expertise of each reviewer for their assigned papers, several other considerations make introducing randomization into the paper assignment desirable: robustness to malicious behavior, the ability to evaluate alternative paper assignments, reviewer diversity, and reviewer anonymity. However, it is unclear in what way one should randomize the paper assignment in order to best satisfy all of these considerations simultaneously. In this work, we present a practical, one-size-fits-all method for randomized paper assignment intended to perform well across different motivations for randomness. We show theoretically and experimentally that our method outperforms currently-deployed methods for randomized paper assignment on several intuitive randomness metrics, demonstrating that the randomized assignments produced by our method are general-purpose.

A One-Size-Fits-All Approach to Improving Randomness in Paper Assignment

Abstract

The assignment of papers to reviewers is a crucial part of the peer review processes of large publication venues, where organizers (e.g., conference program chairs) rely on algorithms to perform automated paper assignment. As such, a major challenge for the organizers of these processes is to specify paper assignment algorithms that find appropriate assignments with respect to various desiderata. Although the main objective when choosing a good paper assignment is to maximize the expertise of each reviewer for their assigned papers, several other considerations make introducing randomization into the paper assignment desirable: robustness to malicious behavior, the ability to evaluate alternative paper assignments, reviewer diversity, and reviewer anonymity. However, it is unclear in what way one should randomize the paper assignment in order to best satisfy all of these considerations simultaneously. In this work, we present a practical, one-size-fits-all method for randomized paper assignment intended to perform well across different motivations for randomness. We show theoretically and experimentally that our method outperforms currently-deployed methods for randomized paper assignment on several intuitive randomness metrics, demonstrating that the randomized assignments produced by our method are general-purpose.

1 Introduction

Peer review is the process in which submissions (such as scientific papers) are evaluated by expert reviewers. It is considered a critical part of the scientific process and is commonly used to determine which papers get published in journals and conferences. For concreteness, we set this work in the academic conference setting, although the approach can be generalized to other settings, such as peer review for grant proposals and peer grading in classrooms. Due to the large scale of modern conferences like NeurIPS and AAAI, conference program chairs work closely with assignment algorithms to assign papers to reviewers automatically. Among many other challenges involved in managing huge numbers of reviewers and submissions, these organizers are faced with the difficult task of balancing various considerations for the paper assignment. Human-friendly automated paper assignment algorithms are thus crucial for helping them find desirable paper assignments.

In a standard paper assignment setting, a set P of n p papers need to be assigned to a set R of n r reviewers. To ensure each paper gets enough reviewers and no reviewer is overloaded with papers, each paper p in P should be assigned to ℓ p reviewers and each reviewer r in R should receive no more than ℓ r papers. An assignment is represented as a binary matrix x in { 0 , 1 } n p × n r , where x p,r = 1 indicates that paper p is assigned to reviewer r . The main objective of paper assignment is usually to maximize the predicted match quality between reviewers and papers [1]. To characterize this,

a similarity matrix S in R n p × n r ≥ 0 is commonly assumed [1-7]. Here, S p,r represents the predicted quality of review from reviewer r for paper p and is generally computed from various sources [8]: reviewer and paper subject areas, reviewer-selected bids, and textual similarity between the paper and the reviewer's past work [1, 9-12]. Then, the quality of an assignment can be defined as the total similarity of all assigned paper reviewer pairs, i.e., Quality( x ) = ∑ p,r x p,r S p,r . One standard approach for computing a paper assignment is to maximize quality [1, 4-7, 13] (which we will refer to as the maximum-quality assignment). Variants of this approach have been widely used by existing conferences, such as NeurIPS, AAAI, and ICML [8].

While the deterministic maximum-quality assignment is the most common, there are strong reasons to introduce randomness into paper assignment - that is, to determine a probability distribution over feasible deterministic assignments and sample one assignment from the distribution. Specifically, randomized paper assignments are beneficial due to the following motivations:

Motivation 1: Robustness to malicious behavior. Several computer science conferences have uncovered 'collusion rings' of reviewers and authors [14, 15], in which the reviewers aim to get assigned to the authors' papers in order to give them good reviews without considering their merits. By manipulating their stated expertise and interest (e.g., in the 'paper bidding' process), these reviewers can cause the assignment algorithm to believe that their match quality with the targeted papers is very high. In other cases, reviewers may target assignment to a paper with the aim of giving it an unfair negative review [16-18]. Randomization can reduce the probability that a malicious reviewer achieves assignment to a target paper.

Motivation 2: Evaluation of alternative assignments. Accurate reviewer-paper similarity scores are fundamental for automated paper assignment algorithms. Despite this, these scores are currently computed using various different methods by different conferences [19, 20], with no obvious way to tell beforehand which method produces the best-quality reviews. However, after deploying an assignment, it is possible to examine the resulting reviews to counterfactually evaluate the review quality produced by another method of similarity computation that is not deployed. Specifically, using techniques for off-policy evaluation [21], the randomness of the deployed paper assignment can be utilized to estimate the review quality of another non-deployed assignment. The variance of the estimation depends on the overlap in assignment probability between the deployed (randomized) assignment and the non-deployed alternative assignments of interest. Such an evaluation can then inform program chairs on how similarities should be computed (or how other algorithmic choices should be made) in the future.

Motivation 3: Reviewer diversity. As each paper is evaluated by multiple reviewers, it is often desirable to assign a set of reviewers with diverse perspectives or areas of expertise. However, since maximum-quality assignments compute only a holistic score to represent the expertise of each reviewer-paper pair, they do not consider this factor. Randomization can increase diversity by spreading out assignment probability among a larger set of high-expertise reviewers.

Motivation 4: Reviewer anonymity. In peer review, reviewer identities are hidden from authors so that authors cannot retaliate for negative reviews. As a result, conferences are generally reluctant to release paper assignment data since authors may be able to deduce the identities of their reviewers (even if reviewer names and other information are hidden). By sufficiently randomizing the assignment, conferences can make it difficult for authors to identify any reviewer on their paper with high probability from the assignment data.

Despite the significance of randomness in paper assignment, there is very limited prior work that looks into computing randomized assignments. A notable exception is [22], which proposed an algorithm for computing randomized paper assignments: Probability Limited Randomized Assignment (PLRA) . Formally, it represents a randomized assignment as a matrix x in [0 , 1] n p × n r , where x p,r denotes the marginal probability that paper p is assigned to reviewer r . PLRA computes a randomized assignment via the following linear program (LP), defined for a given parameter Q ∈ [0 , 1] as:

where Quality( x ) for a randomized assignment x is the expected total similarity of the assignment. A deterministic assignment can then be sampled, using the fact that any feasible randomized assignment x can be implemented as a distribution over feasible deterministic assignments [22, 23].

PLRA is primarily concerned with the first motivation for randomization: robustness to malicious behavior. By limiting each entry of x to be at most Q , PLRA guarantees that any malicious reviewer aiming to be assigned to a target paper has at most probability Q to succeed, even if the reviewer and paper are chosen adversarially. The hyperparameter Q can be adjusted to balance the loss in quality and level of randomization. PLRA has been deployed in multiple iterations of the AAAI conference [8] and is implemented at the popular conference management system OpenReview.net [24].

However, PLRA does not fully solve the problem of randomized paper assignment. In particular, PLRA is specific to one metric of randomness: the maximum assignment probability across all paperreviewer pairs. As a result, it is not clear how well PLRA aligns with motivations for randomization other than robustness to malicious behavior. Moreover, PLRA does not distinguish between different solutions with the same quality and maximum assignment probability. This means that it often loses opportunities to add additional randomness since it neglects to consider non-maximum assignment probabilities. One way to remedy this issue is to allow different values of Q to be set for each pair ( p, r ) , as in the original formulation of [22]. However, this level of flexibility makes it a significant burden for program chairs to manually choose appropriate Q values, hindering the usability of the algorithm. As a result, only the singleQ version stated above has been deployed in practice.

In this work, we address the problem of randomizing paper assignment by looking for a simple and practical method of achieving general-purpose randomized paper assignments. We consider various intuitive randomness metrics that are relevant to all above motivations but not overly specific to a particular problem formulation, and aim to provide a method that performs well across these metrics. In this way, we can provide a method for randomized paper assignment that conference program chairs can easily deploy without needing to precisely specify objectives or hyperparameters.

More specifically, we make the following contributions in this work. (1). We define several metrics to measure the extent to which the randomization in a randomized paper assignment satisfies the stated motivations (Section 3). (2). We propose Perturbed Maximization (PM), a practical algorithm for randomized paper assignment that does not rely on any specific formulation of the stated motivations (Section 4). While our algorithm can be implemented using a standard convex optimization solver, we additionally propose an approximate implementation that is computationally cheaper. (3). We provide theoretical results showing that PM provably outperforms PLRA on two classes of structured similarity scores (Section 5). (4). We extensively evaluate our algorithm via experiments using two realistic datasets of similarity scores from AAMAS 2015 and ICLR 2018 (Section 6). PM simultaneously performs well on all defined randomness metrics while sacrificing a small amount of quality as compared to the optimal non-randomized assignment. Additionally, our experiments show that PM achieves good performance when hyperparameters are set based only on the desired assignment quality, ensuring that it is simple for program chairs to deploy in practice.

2 Related Work

This work follows a recent area of research in computer science on paper assignment algorithms for peer review. Building on the standard approach of maximum-similarity assignment, algorithms have been proposed to handle various additional considerations in the paper assignment: the fairness across papers [2], the seniority of reviewers [20], review processes with multiple phases [3], strategyproofness [25, 26], and many others [8]. We note here specifically two relevant lines of work.

One motivation for our randomized assignment algorithm is to provide robustness to malicious reviewers, a problem considered by several past works [27, 28]. Although our work most closely relates to the randomized paper assignment algorithm proposed in [22], other non-randomized approaches to the problem exist. Leyton-Brown et al. [20] describe the paper assignment process used at AAAI 2021, which included several additional soft constraints intended to curb the possibility of reviewer-author collusion rings. Wu et al. [29] propose fitting a model of reviewer bidding in order to smooth out irregular bids. Boehmer et al. [30] consider computing paper assignments without short-length cycles in order to prevent quid-pro-quo agreements between reviewers.

Randomization in the paper assignment process has also been used for evaluating different assignment policies. Traditionally, conferences will sometimes run randomized controlled experiments in order to test policy changes, where the randomization is incorporated in the assignment of reviewers to different experimental conditions. For example, WSDM 2017 randomly separated reviewers into single- and double-blind conditions in order to evaluate the benefits of double-blind reviewing [31]; other notable experiments include NeurIPS 2014 [32, 33], ICML 2020 [34], and NeurIPS 2021 [35]. In contrast, recent work by Saveski et al. [21] proposes a method for evaluating alternative assignment policies by leveraging randomization in the paper assignment itself, such as the randomized paper assignments of [22]. By introducing additional randomness at a low cost, our algorithm provides an improved basis for the methods of [21] and potential future counterfactual policy evaluation methods.

3 Metrics for Randomness and Problem Statement

In Section 1, we introduced several motivations for choosing randomized paper assignments: robustness to malicious behavior, evaluation of alternative assignments, reviewer diversity, and reviewer anonymity. These indicate that assignment quality is not the only objective that should be considered when choosing a paper assignment. In this section, we propose several metrics to characterize the extent to which the randomness in a paper assignment is practically useful.

One randomness metric considered by PLRA is maximum probability , defined as Maxprob( x ) = max p,r { x p,r } . PLRA controls Maxprob in order to trade off between quality and randomness, and thus achieves the greatest possible quality for a fixed level of Maxprob . However, as the following example shows, this metric alone is not sufficient to fully characterize the randomness of a paper assignment since it ignores the structure of the assignment with non-maximum probability.

p

1

p

2

p

3

p

4

p

5

r

1

r

2

r

3

r

4

r

5

Subject Area A

Subject Area B

1

3

1

2

(a) The ideal assignment

p

1

p

2

p

3

p

4

p

5

r

1

r

2

r

3

r

4

r

5

Subject Area A

Subject Area B

1

2

1

2

(b) PLRA with

Q

=

1

2

Fig. 1 depicts a mini-conference consisting of 2 subject areas. Subject area A contains 3 papers and 3 reviewers, whereas subject area B contains 2 of each. The similarities between reviewers and papers within the same subject area are 1 and similarities across subject areas are 0. The constraints are ℓ p = ℓ r = 1 , i.e., one-to-one assignment. Intuitively, the best randomized assignment matches papers and reviewers uniformly at random within each subject area, like Fig. 1a, since this does not sacrifice any quality. However, PLRA fails to give out this ideal assignment regardless of the hyperparameter choice. Specifically, Q ≥ 1 / 2 is required for PLRA to get a maximum-quality assignment, but when Q ≥ 1 / 2 , PLRA considers Fig. 1a and Fig. 1b to be equivalent (in terms of objective value). In fact, infinitely many solutions with the same optimal quality are considered equivalent by PLRA, and current LP solvers tend to return Fig. 1b as it is a vertex solution when Q = 1 / 2 . In essence, PLRA is leaving 'free randomness on the table,' with many practical implications.

Consider the problem of mitigating malicious behavior (Motivation 1), and compare the assignments in Figs. 1a and 1b. Although they appear to have the same Maxprob , when we only look at subject area A, the Maxprob of Fig. 1a is lower than that of Fig. 1b. This indicates that Fig. 1a is more robust to malicious behavior within subject area A. Therefore, Fig. 1a is more desirable.

Moreover, as introduced in Motivation 2, randomness can also be leveraged to evaluate alternative paper assignments using observations of the review quality from a deployed, randomized assignment. These techniques rely on a 'positivity' assumption: paper-reviewer pairs assigned in the alternative assignment must be given non-zero probability in the deployed assignment. Thus, if we spread assignment probability more uniformly among more reviewer-paper pairs, the resulting data can be

used to evaluate more varied strategies with tighter bounds. In this sense, the assignment in Fig. 1a will allow us to better estimate the quality of different paper assignments within subject area A.

The failure of PLRA on such a simple example shows that Maxprob alone is an inadequate metric. Thus, we need other metrics to distinguish between assignments like Fig. 1a and Fig. 1b. We therefore propose, in addition to Maxprob , a set of new randomness metrics to capture the neglected low-probability structure of an assignment. Under each of these metrics, a uniform assignment is considered 'more random' than any other assignment, thus distinguishing Fig. 1a from Fig. 1b.

Average maximum probability: AvgMaxp( x ) = 1 n p ∑ p max r { x p,r } . With respect to the motivation of preventing malicious behavior, this randomness metric corresponds to the case when a target paper is randomly chosen and the reviewer targeting assignment to that paper is adversarially chosen. By minimizing average maximum probability, we will limit the success probability of manipulation in that case.

Support size: Support( x ) = ∑ p,r I [ x p,r > 0] . Support size directly relates to the 'positivity' assumption introduced above. As there may be many alternative paper assignments of interest, maximizing the support size effectively maximizes the quality of estimation across them.

Entropy: Entropy( x ) = -∑ p,r x p,r ln( x p,r ) . In information theory, entropy characterizes the uncertainty of a random variable. By maximizing entropy, we maximize the uncertainty of our assignment, corresponding to the idea of maximizing randomness. Note that strictly speaking, assignment x is not a probability distribution, so this definition is a generalization.

L2 norm: L2Norm( x ) = √∑ p,r x 2 p,r . To prevent manipulation, PLRA limits the assigned probability of each pair to be at most a specified value Q . L2 norm relaxes this constraint to a soft one: a higher probability results in a larger loss. Note that a uniformly random assignment will always have the smallest L2 norm, and so minimizing L2 norm pushes the assignment towards the uniform assignment.

The combination of these metrics more comprehensively captures the impact of randomness on the motivations from Section 1. Moreover, they do so without requiring a specific problem formulation for each motivation (e.g., an assumption on the behavior of malicious reviewers, a list of the alternative assignments of interest), which are impractical or infeasible to accurately specify in practice.

Problem statement. For an input instance ( n p , n r , ℓ p , ℓ r , S ) , we want to find an algorithm achieving a good trade-off between quality and randomness. Specifically, let the maximum possible quality be M . For a given lower bound of assignment quality ηM ( η ∈ [0 , 1]) , we want the algorithm to produce an assignment x with lower Maxprob( x ) , AvgMaxp( x ) , L2Norm( x ) and higher Entropy( x ) , Support( x ) , i.e., a good Pareto-frontier of quality and randomness.

4 Perturbed Maximization

In this section, we present our proposed algorithm for randomized paper assignment. To describe it, we first present some definitions.

Definition 4.1 (Perturbation Function) . A function f : [0 , 1] → [0 , 1] is a perturbation function if (i) f (0) = 0 , (ii) f ′ exists, (iii) f is non-decreasing on [0 , 1] and (iv) f is concave on [0 , 1] .

Definition 4.2 (Perturbed Quality) . For an assignment x , its perturbed quality with respect to perturbation function f on instance ( n p , n r , ℓ p , ℓ r , S ) is PQuality f ( x ) = ∑ p,r S p,r · f ( x p,r ) .

The definition of perturbed quality incorporates the intuition from the motivating example in the previous section. As function f is concave, the marginal increase of PQuality f ( x ) from increasing a specific entry x p,r is diminishing as x p,r grows. Consequently, the assignment of Fig. 1a will have a higher perturbed quality than that of Fig. 1b. This naturally gives our new algorithm, Perturbed Maximization (PM) . For a given parameter Q ∈ [0 , 1] and a perturbation function f :

Note that PM is a class of algorithms induced by different perturbation functions. When the perturbation function f is chosen to be a linear function, PM becomes PLRA. In the main experiments of this paper, two specific perturbation functions are considered: (1). Exponential Perturbation Function: f ( x ) = 1 -e -αx where α ∈ (0 , + ∞ ) and (2). Quadratic Perturbation Function: f ( x ) = x -βx 2 where β ∈ [0 , 1] . Respectively, we will denote PM with f ( x ) = 1 -e -αx and f ( x ) = x -βx 2 as PM-Exponential (PM-E) and PM-Quadratic (PM-Q) in the rest of the paper.

In Section 6, we will empirically show that under our settings of hyperparameters, PM-E and PM-Q have almost identical performances on every metric we consider, which suggests that the specific form of the perturbation function has limited impact as long as it is strictly concave. Therefore, there is likely no need to consider many different types of perturbation functions.

To analyze PM, first notice that the concaveness of function f guarantees that the optimization program is concave. Therefore, we can use standard concave optimization methods like gradient ascent or the ellipsoid method to solve the program in polynomial time. In most of the experiments of this paper, we will use Gurobi [36], a well-known commercial solver, to solve PM. While solving PM as a general concave optimization problem is conceptually convenient, doing so also incurs a high time complexity as there are n p · n r variables. To further speed up the execution of PM, we propose Algorithm 1, a network-flow-based approximation of PM.

Algorithm 1 : Network-Flow-Based Approximation of PM

For a given parameter Q ∈ [0 , 1] , a perturbation function f and a precision w ∈ N + :

Construct a graph G with a source s , a sink t .

For paper p , add a vertex v p and an edge s → v p with capacity ℓ p · w and cost 0 .

For reviewer r , add a vertex v r and an edge v r → t with capacity ℓ r · w and cost 0 .

For a reviewer-paper pair ( p, r ) , add ⌊ Q · w ⌋ edges v p → v r . The i -th edge has capacity 1 and cost S p,r · [ f ( i w ) -f ( i -1 w )] . Let this set of edges be E p,r .

Run maximum cost maximum flow algorithm [37] on G . Define the assignment x such that x p,r = [ The total flow on edges in E p,r ] /w .

Enumerate all pairs ( p, r ) in any order and increase x p,r by min { Q -x p,r , ℓ p -∑ r x p,r , ℓ r -∑ p x p,r } . If ∑ p,r x p,r = n p · ℓ p , return x . Otherwise, return infeasible.

At a high level, Algorithm 1 uses a piecewise linear function with w pieces to approximate the concave function f ( x ) and solves the approximated objective with maximum cost maximum flow. As we increase the precision w , the approximation becomes more accurate, but the running time of the algorithm also scales up. Formally, we have the following Theorem 1.

Theorem 1. Let w ∈ N + be the precision in Algorithm 1, and OPT be the optimal perturbed quality.

(a) Algorithm 1 runs in O ( w · ℓ p · n p 2 · n r ) time.

(b) Algorithm 1 returns an assignment with perturbed quality ALG ≥ OPT -f ( 1 w ) ∑ p,r S p,r .

The proof of Theorem 1 is deferred to Appendix B.1.

Consider the running time of Algorithm 1 given in Theorem 1 (a). If we directly model PM as an optimization problem, the number of variables will be n = n p · n r . The state-of-the-art algorithm for solving a linear program with n variables to high accuracy has a time complexity of O ∗ ( n 2 . 38 log n ) [38]. In contrast, for fixed w , Algorithm 1 works in O ( n 2 ) time since ℓ p ≤ n r . Thus, Algorithm 1 has a better time complexity than directly solving PM as an optimization program even if the objective is linear. For non-linear perturbation functions, the time complexity of Algorithm 1 remains the same while the complexity of solving the optimization program increases. Moreover, by Theorem 1 (b), we can see that as w increases, the approximated perturbed quality ALG approaches OPT , formalizing the intuition that as we increase the precision w , the approximation becomes more accurate. In Section 6, we empirically evaluate the running time and the approximation accuracy of Algorithm 1. We find with w = 10 , Algorithm 1 produces decently-accurate approximations on our datasets.

5 Theoretical Analysis

In this section, we provide two theorems showing that PM provably outperforms PLRA on a general class of input instances. We start with the simpler one inspired by the example in Fig. 1. Let PLRA( Q ) and PM( Q,f ) be the set of possible solutions of PLRA and PM respectively.

Definition 5.1 (Blockwise Dominant Matrix) . Asimilarity matrix S ∈ R n p × n r ≥ 0 is blockwise dominant with block identity A ∈ R k × k ≥ 0 and block sizes { p 1 , . . . , p k } , { r 1 , . . . , r k } if

̸

where A i,i > A i,j , ∀ i = j and 1 p × r is a p × r matrix with all entries being 1 . Moreover, define the dominance factor of A as Dom( A ) = sup { α | A i,i ≥ α · A i,j , ∀ i = j } .

̸

̸

Remark. Like Fig. 1, a blockwise dominant similarity matrix models a conference with k subject areas where the i -th subject area has p i papers and r i reviewers. The similarity between a paper and a reviewer is determined only by their subject areas, and a paper has highest similarity with a reviewer in the same subject area. Note that Dom( A ) > 1 as A i,i > A i,j , ∀ i = j .

Theorem 2. For an input instance ( n p , n r , ℓ p , ℓ r , S ) , where S is blockwise dominant with block identity A ∈ R k × k ≥ 0 and block sizes { p 1 , . . . , p k } , { r 1 , . . . , r k } , assume (i) { r 1 , . . . , r k } are not all equal, (ii) p i · ℓ p ≤ r i · ℓ r ∀ i ∈ { 1 , . . . , k } and (iii) Q · r i ≥ ℓ p ∀ i ∈ { 1 , . . . , k } . Let f be a strictly concave perturbation function and f ′ (0) < Dom( A ) f ′ (1) . PM with f ( x ) as the perturbation function (weakly) dominates PLRA in quality and all randomness metrics. Formally,

(a) ∀ x ∈ PM( Q,f ) , ∀ y ∈ PLRA( Q ) ,

Quality( x ) ≥ Quality( y ) , Maxprob( x ) ≤ Maxprob( y ) , AvgMaxp( x ) ≤ AvgMaxp( y ) , L2Norm( x ) ≤ L2Norm( y ) , Entropy( x ) ≥ Entropy( y ) , Support( x ) ≥ Support( y ) .

(b) ∀ x ∈ PM( Q,f ) , ∃ y ∈ PLRA( Q ) ,

AvgMaxp( x ) < AvgMaxp( y ) , L2Norm( x ) < L2Norm( y ) , Entropy( x ) > Entropy( y ) .

The proof of Theorem 2 follows the same intuition as the example in Fig. 1. The assumptions guarantee that an assignment like Fig. 1a, i.e., uniformly matching papers to reviewers within the same subject area, is feasible. Details of the proof are deferred to Appendix B.2. At a high level, Theorem 2 shows that with a slight restriction on the perturbation function, PM provably performs better than PLRA on blockwise dominant similarity matrices.

Theorem 2 requires a strict restriction on S 's structure. In our next theorem, we remove the restriction, stating that PM also outperforms PLRA on a random similarity matrix with high probability.

Theorem 3. For an input instance ( n p , n r , ℓ p , ℓ r , S ) , where each entry of S is i.i.d . sampled from { v 1 , . . . , v k } (0 < v 1 < · · · < v k ) uniformly, assume (i) k ≤ 1 c · n p , (ii) ℓ r ≥ c · ln( n r ) , (iii) 2 · n p · ℓ p ≤ n r · ℓ r and (iv) Q · ( n r -1) ≥ ℓ p . Let f be a strictly concave perturbation function and f ′ (0) < v i v i -1 f ′ (1) , ∀ i ∈ { 2 , . . . , k } . With probability 1 -e -Ω( c ) , PM with f ( x ) as the perturbation function (weakly) dominates PLRA in quality and all randomness metrics. Formally,

(a) ∀ x ∈ PM( Q,f ) , ∀ y ∈ PLRA( Q )

Quality( x ) ≥ Quality( y ) , Maxprob( x ) ≤ Maxprob( y ) , AvgMaxp( x ) ≤ AvgMaxp( y ) L2Norm( x ) ≤ L2Norm( y ) , Entropy( x ) ≥ Entropy( y ) , Support( x ) ≥ Support( y ) .

, ,

(b) ∀ x ∈ PM( Q,f ) , ∃ y ∈ PLRA( Q ) ,

Support( x ) > Support( y ) , L2Norm( x ) < L2Norm( y ) , Entropy( x ) > Entropy( y ) .

The complete proof of Theorem 3 is deferred to Appendix B.3. To sketch the proof, we will first relate PLRA and PM to two simpler auxiliary algorithms using a concentration inequality and then

prove the dominance between them. The assumptions (ii) and (iii) are used to connect PM and PLRA with the auxiliary algorithms, while (i) and (iv) are for proving the dominance.

For Theorem 3 to hold on general random similarity matrices, we have made a major assumption (i), which requires that there are not too many distinct levels of similarity scores. Assumption (i) is naturally satisfied in some cases. For example, when the similarities are derived purely from reviewers' bids, the number of levels becomes the number of discrete bid levels; discrete values can also result when computing similarities from the overlap between reviewer- and author-selected subject areas. When the similarity scores are instead computed from continuous values like TPMS scores [1], assumption (i) is usually not satisfied. In Section 6, we show experiments on both discrete and continuous similarities to evaluate the effect of assumption (i).

6 Experiments

Datasets. In this section, we test our algorithm on two realistic datasets. The first dataset is bidding data from the AAMAS 2015 conference [39]. In this dataset, n p = 613 , n r = 201 , and the bidding data has 4 discrete levels: 'yes', 'maybe', 'no' and 'conflict'. We transform these 4 levels to similarities 1 , 1 2 , 1 4 and 0 , as inspired by NeurIPS 2016, in which similarity scores were computed as 2 bid (0 . 5 s text +0 . 5 s subject ) [19]. The transformed dataset satisfies assumption (i) in Theorem 3 as there are only 4 levels. The second dataset contains text-similarity scores recreated from the ICLR 2018 conference with n p = 911 , n r = 2435 [26]. These scores were computed by comparing the text of each paper with the text of each reviewer's past work; we directly use them as the similarity matrix. Assumption (i) is not valid in this dataset as it is continuous-valued. The constraints are set as ℓ p = 3 , ℓ r = 6 for ICLR 2018 as was done in [26] and ℓ p = 3 , ℓ r = 12 for AAMAS 2015 for feasibility. In Appendix A.3, we also test our algorithm on four additional datasets from [39].

Experiment and hyperparameter setting. We implement PLRA and two versions of PM (PM-E and PM-Q) using commercial optimization solver Gurobi 10.0 [36]. For each algorithm on each dataset, we use a principled method (Appendix A.2) to find 8 sets of hyperparameters that produce solutions with at least { 80% , 85% , 90% , 95% , 98% , 99% , 99 . 5% , 100% } of the maximum possible quality. When setting the hyperparameters, we choose a 'slackness' value δ and allow PM to produce solutions with Maxprob at most δ higher than the optimal Maxprob (in exchange for better performance on other metrics). For AAMAS 2015, δ = 0 . 00 and for ICLR 2018, δ = 0 . 02 . We refer to Appendix A.2 for more details. We do not consider assignments with lower than 80% of the maximum quality since such low-quality assignments are unlikely to be deployed in practice. We evaluate the produced assignments on different randomness metrics to draw Figs. 2 to 4. We also implemented Algorithm 1 and tested it with the set of parameters at 95% relative quality to create Table 1. All source code is released at https://github.com/YixuanEvenXu/perturbed-maximization , and all experiments are done on a server with 56 cores and 504G RAM, running Ubuntu 20.04.6.

Comparing Gurobi and Algorithm 1. Table 1 shows that on ICLR 2018, Gurobi takes less wall clock time than Algorithm 1 due to parallelization over the server's 112 cores. However, Algorithm 1 takes less total CPU time (system + user CPU time) and achieves similar performance with Gurobi, which shows that Algorithm 1 provides decently approximated solutions using fewer computation

resources. Due to the space limit, rows in Table 1 have been selected and only results on ICLR 2018 are present. The complete tables and analysis for both datasets can be found in Appendix A.1.

Comparing randomness metrics of PM and PLRA on discrete-valued datasets. As shown in Figs. 2a and 3, on AAMAS 2015, both versions of PM achieve exactly the same performance with PLRA on Maxprob while improving significantly on the other randomness metrics. Recall that PLRA achieves the optimal quality for a given Maxprob . This demonstrates that as predicted by Theorem 3, PM produces solutions that are more generally random while preserving the optimality in Maxprob on discrete-valued datasets where assumption (i) holds. In Appendix A.3, we also test our algorithm on four more datasets from [39] and observe similar results.

0.80

0.85

0.90

0.95

1.00

Relative Quality

0.5

0.6

0.7

0.8

0.9

1.0

Maxprob

PLRA

PM-Q

PM-E

(a) AAMAS 2015 Quality vs Maxprob ( ↓ )

0.80

0.85

0.90

0.95

1.00

Relative Quality

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Maxprob

(b) ICLR 2018

Quality

vs

↓

)

0.80

0.85

0.90

0.95

1.00

Relative Quality

0.4

0.5

0.6

0.7

0.8

0.9

1.0

AvgMax

PLRA

PM-Q

PM-E

(a) AAMAS 2015

Quality

vs

AvgMaxp (

↓

)

0.80

0.85

0.90

0.95

1.00

Relative Quality

22.5

25.0

27.5

30.0

32.5

35.0

37.5

40.0

42.5

L2 Norm

(b) AAMAS 2015

Quality

vs

L2Norm (

↓

)

0.80

0.85

0.90

0.95

1.00

Relative Quality

0

10000

20000

30000

40000

50000

60000

Support

(c) AAMAS 2015 Quality vs Support ( ↑ )

0.80

0.85

0.90

0.95

1.00

Relative Quality

0

500

1000

1500

2000

2500

3000

3500

Entropy

(d) AAMAS 2015

Quality

vs

Entropy (

↑

)

Comparing randomness metrics of PM and PLRA on continuous-valued datasets. The results on ICLR 2018 in Figs. 2b and 4 show that both versions of PM sacrifice some Maxprob as compared to PLRA. The exact amount of this sacrifice is affected by the slackness δ described in Appendix A.2. However, the improved randomness is still significant, though to a lesser extent than on AAMAS 2015. Note that assumption (i) of Theorem 3 does not hold on ICLR 2018. This exhibits the empirical efficacy of PM beyond the theoretical guarantees provided in Section 5.

Comparing randomness metrics of PM-E and PM-Q. In all of Figs. 2 to 4, we can see that no matter what metric and dataset are used, the performances of PM-E and PM-Q are always similar.

0.80

0.85

0.90

0.95

1.00

Relative Quality

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

AvgMax

PLRA

PM-Q

PM-E

(a) ICLR 2018

Quality

vs

AvgMaxp (

↓

)

0.80

0.85

0.90

0.95

1.00

Relative Quality

35.0

37.5

40.0

42.5

45.0

47.5

50.0

52.5

L2 Norm

(b) ICLR 2018

Quality

vs

L2Norm (

↓

)

0.80

0.85

0.90

0.95

1.00

Relative Quality

4000

6000

8000

10000

12000

Support

(c) ICLR 2018

Quality

vs

Support (

↑

)

0.80

0.85

0.90

0.95

1.00

Relative Quality

0

500

1000

1500

2000

2500

3000

Entropy

(d) ICLR 2018

Quality

vs

Entropy (

↑

)

This suggests that the specific type of the perturbation function has limited impact as long as it is strictly concave. Therefore, program chairs do not need to carefully consider this choice in practice.

Additional experiments. Due to the page limit on the main paper, we are not able to present all the experiments we run in this section. We refer to Appendix A for additional experimental results about the network-flow-based approximation (Appendix A.1), hyperparameter tuning (Appendix A.2) and experiments on more datasets (Appendix A.3).

7 Conclusion

We present a general-purpose, practical algorithm for use by conference program chairs in computing randomized paper assignments. We show both theoretically and experimentally that our algorithm significantly improves over the previously deployed randomized assignment algorithm, PLRA. In conferences that currently deploy PLRA, our algorithm can be simply plugged-in in place of it to find a paper assignment with the same quality but with an improved level of randomization, achieving various benefits.

In practice, various aspects of paper assignment other than quality and randomness are considered by program chairs. We refer to Appendix C for discussions about how to optimize a specific metric with our algorithm (Appendix C.1) and how to incorporate some additional constraints from [20] (Appendix C.2). That being said, we do not consider various other aspects and constraints that may be desired by program chairs: e.g., fairness-based objectives [2] or constraints on review cycles [20]. Incorporating these aspects remains an interesting direction for future work.

Broader Impacts. We expect our algorithm to have a mostly positive social impact by assisting program chairs in mitigating malicious behavior, facilitating evaluation of assignments, and increasing reviewer diversity and reviewer anonymity. While deploying a randomized assignment may negatively impact the assignment quality as compared to a deterministic assignment, our work allows conferences to choose the level of randomization that they deem best for their goals.

Acknowledgments and Disclosure of Funding

This work was supported by ONR grant N000142212181 and NSF grant IIS-2200410.

This work was supported in part by the Sloan Research Fellowship.

References

Laurent Charlin and Richard S. Zemel. The Toronto Paper Matching System: An automated paper-reviewer assignment system. In ICML Workshop on Peer Reviewing and Publishing Models , 2013.

Ivan Stelmakh, Nihar B. Shah, and Aarti Singh. PeerReview4All: Fair and accurate reviewer assignment in peer review. In ALT , 2019.

Steven Jecmen, Hanrui Zhang, Ryan Liu, Fei Fang, Vincent Conitzer, and Nihar B Shah. Nearoptimal reviewer splitting in two-phase paper reviewing and conference experiment design. In HCOMP , 2022.

Wenbin Tang, Jie Tang, and Chenhao Tan. Expertise matching via constraint-based optimization. In International Conference on Web Intelligence and Intelligent Agent Technology , 2010.

Peter A. Flach, Sebastian Spiegler, Bruno Golénia, Simon Price, John Guiver, Ralf Herbrich, Thore Graepel, and Mohammed J. Zaki. Novel tools to streamline the conference review process: Experiences from SIGKDD'09. SIGKDD Explorations Newsletter , 2010.

Camillo J. Taylor. On the optimal assignment of conference papers to reviewers. Technical report, Department of Computer and Information Science, University of Pennsylvania, 2008.

Laurent Charlin, Richard S. Zemel, and Craig Boutilier. A framework for optimizing paper matching. In UAI , 2011.

Nihar B Shah. Challenges, experiments, and computational solutions in peer review. Communications of the ACM , 2022.

David Mimno and Andrew McCallum. Expertise modeling for matching papers with reviewers. In KDD , 2007.

Xiang Liu, Torsten Suel, and Nasir Memon. A robust model for paper reviewer assignment. In RecSys , 2014.

Marko A. Rodriguez and Johan Bollen. An algorithm to determine peer-reviewers. In CIKM , 2008.

Hong Diep Tran, Guillaume Cabanac, and Gilles Hubert. Expert suggestion for conference program committees. In RCIS , 2017.

Judy Goldsmith and Robert H. Sloan. The AI conference paper assignment problem. In AAAI Workshop , 2007.

T. N. Vijaykumar. Potential organized fraud in ACM/IEEE computer architecture conferences. https://medium.com/@tnvijayk/potential-organized-fraud-in-acmieee-computer-architecture-conferences-ccd61169370d , 2020. Accessed February 1, 2023.

Michael Littman. Collusion rings threaten the integrity of computer science research. Communications of the ACM , 2021.

Jef Akst. I hate your paper. Many say the peer review system is broken. Here's how some journals are trying to fix it. The Scientist , 2010.

Edward F. Barroga. Safeguarding the integrity of science communication by restraining 'rational cheating' in peer review. Journal of Korean Medical Science , 2014.

Mario Paolucci and Francisco Grimaldo. Mechanism change in a simulation of peer review: From junk support to elitism. Scientometrics , 2014.

Nihar B Shah, Behzad Tabibian, Krikamol Muandet, Isabelle Guyon, and Ulrike Von Luxburg. Design and analysis of the nips 2016 review process. Journal of Machine Learning Research , 2018.

Kevin Leyton-Brown, Yatin Nandwani, Hedayat Zarkoob, Chris Cameron, Neil Newman, Dinesh Raghu, et al. Matching papers and reviewers at large conferences. arXiv preprint arXiv:2202.12273 , 2022.

Martin Saveski, Steven Jecmen, Nihar Shah, and Johan Ugander. Counterfactual evaluation of peer review assignment strategies in computer science and artificial intelligence. In NeurIPS , 2023.

Steven Jecmen, Hanrui Zhang, Ryan Liu, Nihar Shah, Vincent Conitzer, and Fei Fang. Mitigating manipulation in peer review via randomized reviewer assignments. In NeurIPS , 2020.

Eric Budish, Yeon-Koo Che, Fuhito Kojima, and Paul Milgrom. Implementing random assignments: A generalization of the Birkhoff-von Neumann theorem. In Cowles Summer Conference , 2009.

OpenReview.net. OpenReview matcher. https://github.com/openreview/openreviewmatcher , 2023. Accessed April 2023.

Komal Dhull, Steven Jecmen, Pravesh Kothari, and Nihar B Shah. Strategyproofing peer assessment via partitioning: The price in terms of evaluators' expertise. In HCOMP , 2022.

Yichong Xu, Han Zhao, Xiaofei Shi, Jeremy Zhang, and Nihar B Shah. On strategyproof conference peer review. In IJCAI , 2019.

Steven Jecmen, Nihar B Shah, Fei Fang, and Vincent Conitzer. Tradeoffs in preventing manipulation in paper bidding for reviewer assignment. In ML Evaluation Standards Workshop at ICLR , 2022.

Steven Jecmen, Minji Yoon, Vincent Conitzer, Nihar B. Shah, and Fei Fang. A dataset on malicious paper bidding in peer review. In WWW , 2023.

Ruihan Wu, Chuan Guo, Felix Wu, Rahul Kidambi, Laurens Van Der Maaten, and Kilian Weinberger. Making paper reviewing robust to bid manipulation attacks. In ICML , 2021.

Niclas Boehmer, Robert Bredereck, and André Nichterlein. Combating collusion rings is hard but possible. In AAAI , 2022.

Andrew Tomkins, Min Zhang, and William D. Heavlin. Reviewer bias in single- versus double-blind peer review. Proceedings of the National Academy of Sciences , 2017.

Neil D. Lawrence. The NIPS experiment. https://inverseprobability.com/2014/12/ 16/the-nips-experiment , 2014. Accessed February 1, 2023.

Eric Price. The NIPS experiment. http://blog.mrtz.org/2014/12/15/the-nipsexperiment.html , 2014. Accessed February 1, 2023.

Ivan Stelmakh, Charvi Rastogi, Nihar B Shah, Aarti Singh, and Hal Daumé III. A large scale randomized controlled trial on herding in peer-review discussions. PLoS ONE , 2023.

Alina Beygelzimer, Yann Dauphin, Percy Liang, and Jennifer Wortman Vaughan. The NeurIPS 2021 consistency experiment. https://blog.neurips.cc/2021/12/08/the-neurips2021-consistency-experiment/ , 2021. Accessed February 1, 2023.

Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2023.

Jack Edmonds and Richard M Karp. Theoretical improvements in algorithmic efficiency for network flow problems. Journal of the ACM (JACM) , 1972.

Michael B Cohen, Yin Tat Lee, and Zhao Song. Solving linear programs in the current matrix multiplication time. Journal of the ACM (JACM) , 2021.

Nicholas Mattei and Toby Walsh. Preflib: A library for preferences http://www.preflib.org. In ADT , 2013.

A Additional Experiments

A.1 Network-Flow-Based Approximation

In Section 6, we presented Table 1, which is a shortened version of the experimental results comparing Gurobi and Algorithm 1. We now present the complete experimental results on both datasets in Tables 2 and 3 as well as the complete analysis in this section.

Comparing the running time of Gurobi and Algorithm 1. As shown in Tables 2 and 3, Gurobi generally runs faster than Algorithm 1 in wall clock time. This is because it is a well-written commercial software that is able to parallelize over multiple cores on our server. However, the sum of user and system CPU time is smaller for Algorithm 1, which shows that Algorithm 1 uses fewer computation resources than Gurobi. Moreover, note that Gurobi takes significantly longer to solve PM-E than PM-Q, while Algorithm 1 is not affected. The reason for this is Gurobi only supports quadratic objective functions primitively. Therefore, to solve exponential objective functions, iterative approximation methods have to be used. In contrast, Algorithm 1 can support an arbitrary perturbation function without sacrificing running time.

Comparing the solution qualities of Gurobi and Algorithm 1. In the rows about Quality and PQuality in Tables 2 and 3, we can see that Algorithm 1 has almost the same Quality as Gurobi. The PQuality of Algorithm 1 is also close to Gurobi's. This shows that with w = 10 , Algorithm 1 approximates PM in solution quality well, validating Theorem 1.

Comparing the randomness metrics of Gurobi and Algorithm 1. As shown in Table 2, for randomness metrics Maxprob , AvgMaxp , and L2Norm , the approximated solution by Algorithm 1 has a similar or identical performance to Gurobi on AAMAS 2015. For Support and Entropy , the approximated solution is significantly worse than Gurobi's, but it is still significantly better than PLRA's solution. As presented in Table 3, on ICLR 2018, the performances of Algorithm 1 and Gurobi on all randomness metrics are relatively close. This shows that Algorithm 1 mostly preserves PM's performance on randomness metrics. Although on discrete-valued datasets like AAMAS 2015, Support and Entropy can be affected, they are still better than PLRA's solution.

A.2 Hyperparameter Tuning

In this section, we will introduce the way in which we do hyperparameter tuning for PM. We start with a lemma that shows the monotonicity of the solution quality of PM-Q with respect to β .

Lemma A.1. For an input instance ( n p , n r , ℓ p , ℓ r , S ) , let the solution of PM with f 1 ( x ) = x -β 1 x 2 be x 1 and the solution of PM with f 2 ( x ) = x -β 2 x 2 be x 2 where β 1 , β 2 ∈ [0 , 1] . Then

As x 1 maximizes PQuality f 1 ( x ) and x 2 maximizes PQuality f 2 ( x ) , we have

Subtracting (2) from (1), and using β 2 > β 1 , we have

Then, from (1), we know that

Therefore, Quality( x 1 ) ≥ Quality( x 2 ) .

With Lemma A.1, for a minimum requirement of quality Quality M , we can use binary search to find the largest β for PM-Q such that PM-Q with f ( x ) = x -βx 2 gives an assignment with quality ≥ Quality M . Formally, consider the following Algorithm 2.

Algorithm 2 : Hyperparameter Tuning for PM-Q

For a given minimum quality requirement Quality M and a slackness δ :

Use binary search to find Q PLRA , the smallest Q with which PLRA gives a solution with quality ≥ Quality M .

Use binary search to find β max , the largest β such that PM-Q with Q = Q PLRA + δ and f ( x ) = x -β 2 gives a solution with quality ≥ Quality M .

Finalize the hyperparameters as ( Q,β ) = (min { Q PLRA + δ, 1 } , β max ) .

Algorithm 2 maximizes the parameter β while ensuring the produced assignment with the set of hyperparameters ( Q,β ) has quality at least Quality M and Maxprob at most Q PLRA + δ . Note that any solution with quality ≥ Quality M has a Maxprob ≥ Q PLRA . Intuitively, Algorithm 2 is

trying to maximize the randomness while obtaining the minimum required quality and near-optimal Maxprob given the quality constraint.

Analogously, we use a similar method for tuning the hyperparameters in PM-E.

Algorithm 3 : Hyperparameter Tuning for PM-E

For a given minimum quality requirement Quality M and a slackness δ :

Use binary search to find Q PLRA , the smallest Q with which PLRA gives a solution with quality ≥ Quality M .

Use binary search to find α max , the largest α such that PM-E with Q = Q PLRA + δ and f ( x ) = 1 -e -αx gives a solution with quality ≥ Quality M .

Finalize the hyperparameters as ( Q,α ) = (min { Q PLRA + δ, 1 } , α max ) .

Although we similarly apply binary search to PM-E as we have done to PM-Q, the monotonicity condition of Lemma A.1 does not hold in general for PM-E. For instance, consider the following Example 1. If we run PM-E with different α and Q = 1 , the solution quality is not monotonic with respect to α . The results are shown in Fig. 5.

Example 1. In this example, n p = n r = 3 , ℓ p = ℓ r = 1 and

0.0

2.5

5.0

7.5 10.012.515.017.520.0

Alpha

0.993

0.994

0.995

0.996

0.997

0.998

0.999

1.000

Relative Quality

Nevertheless, such examples are rare in practice. We have examined thousands of examples to find Example 1. In fact, the monotonicity of solution quality with respect to α can still be observed empirically in the datasets we used. Therefore, for the purpose of our experiments in Section 6, we will still use Algorithm 3 to tune the hyperparameters of PM-E.

In Algorithms 2 and 3, there are still two parameters to be fixed, namely δ and Quality M . In Section 6, Quality M was already specified as { 80% , 85% , 90% , 95% , 98% , 99% , 99 . 5% , 100% } of the maximum possible quality. It remains to specify δ . For experiments in Section 6, we used δ = 0 for AAMAS 2015 and δ = 0 . 02 for ICLR 2018. In the rest of the section, we will show experiments about the performances of PM-Q and PM-E with different δ values to justify our choice. In practice, conference program chairs can simply choose a small value for the slackness, such as δ = 0 . 02 .

0.00

0.04

0.08

0.12

0.16

0.20

Delta

0.74

0.75

0.76

0.77

0.78

0.79

0.80

AvgMax

(a) AAMAS 2015

δ

vs

AvgMaxp (

↓

)

0.00

0.04

0.08

0.12

0.16

0.20

Delta

31

32

33

34

35

36

37

L2 Norm

PLRA

PM-Q

PM-E

PM-Q Elbow

PM-E Elbow

(b) AAMAS 2015

δ

vs

L2Norm (

↓

)

0.00

0.04

0.08

0.12

0.16

0.20

Delta

5000

10000

15000

20000

25000

30000

Support

(c) AAMAS 2015

δ

vs

Support (

↑

)

0.00

0.04

0.08

0.12

0.16

0.20

Delta

500

750

1000

1250

1500

1750

2000

2250

Entropy

(d) AAMAS 2015

δ

vs

Entropy (

↑

)

0.00

0.02

0.04

0.06

0.08

0.10

Delta

0.82

0.83

0.84

0.85

0.86

0.87

0.88

0.89

0.90

AvgMax

(a) ICLR 2018

δ

vs

AvgMaxp (

↓

)

0.00

0.02

0.04

0.06

0.08

0.10

Delta

40

42

44

46

48

L2 Norm

PLRA

PM-Q

PM-E

PM-Q Elbow

PM-E Elbow

(b) ICLR 2018

δ

vs

L2Norm (

↓

)

0.00

0.02

0.04

0.06

0.08

0.10

Delta

4000

6000

8000

10000

12000

Support

(c) ICLR 2018

δ

vs

Support (

↑

)

0.00

0.02

0.04

0.06

0.08

0.10

Delta

500

750

1000

1250

1500

1750

2000

2250

2500

Entropy

(d) ICLR 2018

δ

vs

Entropy (

↑

)

As we increase δ , we allow PM-Q and PM-E to find more random solutions (i.e., solutions that perform better on our randomness metrics) at a cost of larger Maxprob . To balance the gain and loss, we choose δ that maximizes a linear combination of them. We call these δ elbow points . Below, we show the choices of δ graphically in Figs. 6 and 7.

As shown in Figs. 6 and 7, on AAMAS 2015, the elbow points of both algorithms in all four metrics are at δ = 0 . Therefore, we should set δ = 0 for it. On ICLR 2018, the elbow points lie in the range δ ∈ [0 . 02 , 0 . 04] . We have chosen δ = 0 . 02 in this range for it.

A.3 Experiments on More Datasets

In this subsection, we include additional experiment results on more 4 datasets. In particular, we include Preflib1 (00039-1 from [39], n p = 54 , n r = 31 ), Preflib2 (00039-2 from [39], n p = 52 , n r = 24 ), Preflib3 (00039-3 from [39], n p = 176 , n r = 146 ) and AAMAS 2016 (00037-2 from [39], n p = 442 , n r = 161 ). All of the four datasets are bidding data consisting of 4 discrete levels: 'yes', 'maybe', 'no' and 'conflict'. We transform these 4 levels to similarities 1 , 1 2 , 1 4 and 0 as done in Section 6. The constraints are set as ℓ p = 3 , ℓ r = 6 for Preflib1 and Preflib3. For Preflib2 and AAMAS 2016, ( ℓ p , ℓ r ) are set as (3 , 7) and (3 , 12) respectively for feasibility.

We use the same experimental setup and hyperparameter setting as in Section 6. The results are shown in Fig. 8. The results of these additional datasets are similar to those on AAMAS 2015 shown in Figs. 2a and 3. In particular, both versions of PM achieve exactly the same performance with PLRA on Maxprob while improving significantly on the other randomness metrics. This shows that the empirical observations in Section 6 are not dataset-specific and generalize to other datasets.

0.80

0.85

0.90

0.95

1.00

Relative Quality

0.5

0.6

0.7

0.8

0.9

1.0

Maxprob

PLRA

PM-Q

PM-E

(a) Preflib1

Quality

vs

↓

)

0.80

0.85

0.90

0.95

1.00

Relative Quality

0.5

0.6

0.7

0.8

0.9

1.0

Maxprob

(b) Preflib2

Quality

vs

↓

)

0.80

0.85

0.90

0.95

1.00

Relative Quality

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Maxprob

(c) Preflib3

Quality

vs

↓

)

0.80

0.85

0.90

0.95

1.00

Relative Quality

0.5

0.6

0.7

0.8

0.9

1.0

Maxprob

(d) AAMAS2016

Quality

vs

↓

)

0.80

0.85

0.90

0.95

1.00

Relative Quality

0.4

0.5

0.6

0.7

0.8

0.9

1.0

AvgMax

(e) Preflib1

Quality

vs

AvgMaxp (

↓

)

0.80

0.85

0.90

0.95

1.00

Relative Quality

0.5

0.6

0.7

0.8

0.9

1.0

AvgMax

(f) Preflib2

Quality

vs

AvgMaxp (

↓

)

0.80

0.85

0.90

0.95

1.00

Relative Quality

0.4

0.5

0.6

0.7

0.8

0.9

1.0

AvgMax

(g) Preflib3

Quality

vs

AvgMaxp (

↓

)

0.80

0.85

0.90

0.95

1.00

Relative Quality

0.5

0.6

0.7

0.8

0.9

1.0

AvgMax

(h) AAMAS2016

Quality

vs

AvgMaxp (

↓

)

0.80

0.85

0.90

0.95

1.00

Relative Quality

7

8

9

10

11

12

13

L2 Norm

(i) Preflib1

Quality

vs

L2Norm (

↓

)

0.80

0.85

0.90

0.95

1.00

Relative Quality

8

9

10

11

12

L2 Norm

(j) Preflib2

Quality

vs

L2Norm (

↓

)

0.80

0.85

0.90

0.95

1.00

Relative Quality

12

14

16

18

20

22

L2 Norm

(k) Preflib3

Quality

vs

L2Norm (

↓

)

0.80

0.85

0.90

0.95

1.00

Relative Quality

20.0

22.5

25.0

27.5

30.0

32.5

35.0

L2 Norm

(l) AAMAS2016

Quality

vs

L2Norm (

↓

)

0.80

0.85

0.90

0.95

1.00

Relative Quality

200

400

600

800

1000

1200

Support

(m) Preflib1

Quality

vs

Support (

↑

)

0.80

0.85

0.90

0.95

1.00

Relative Quality

200

300

400

500

600

700

Support

(n) Preflib2

Quality

vs

Support (

↑

)

0.80

0.85

0.90

0.95

1.00

Relative Quality

0

2000

4000

6000

8000

10000

12000

14000

Support

(o) Preflib3

Quality

vs

Support (

↑

)

0.80

0.85

0.90

0.95

1.00

Relative Quality

0

5000

10000

15000

20000

25000

30000

35000

40000

Support

(p) AAMAS2016

Quality

vs

Support (

↑

)

0.80

0.85

0.90

0.95

1.00

Relative Quality

0

50

100

150

200

250

Entropy

(q) Preflib1

Quality

vs

Entropy (

↑

)

0.80

0.85

0.90

0.95

1.00

Relative Quality

0

25

50

75

100

125

150

175

200

Entropy

(r) Preflib2

Quality

vs

Entropy (

↑

)

0.80

0.85

0.90

0.95

1.00

Relative Quality

0

200

400

600

800

1000

Entropy

(s) Preflib3

Quality

vs

Entropy (

↑

)

0.80

0.85

0.90

0.95

1.00

Relative Quality

0

500

1000

1500

2000

2500

Entropy

(t) AAMAS2016

Quality

vs

Entropy (

↑

)

B Missing Proofs in Sections 4 and 5

B.1 Proof of Theorem 1

Theorem 1. Let w ∈ N + be the precision in Algorithm 1, and OPT be the optimal perturbed quality.

(a) Algorithm 1 runs in O ( w · ℓ p · n p 2 · n r ) time.

(b) Algorithm 1 returns an assignment with perturbed quality ALG ≥ OPT -f ( 1 w ) ∑ p,r S p,r .

Proof of Theorem 1: For (a), consider implementing the maximum cost flow using Edmonds & Karp's Algorithm B in [37]. In Edmonds & Karp's algorithm, we will compute the shortest path on the residual graph each time we find an augmenting path. The number of times we augment is bounded by O ( w · ℓ p · n p ) . For the shortest path part, Edmonds & Karp's algorithm applies a node potential trick that ensures the edge weights are non-negative throughout the execution, so that we can use Dijkstra's algorithm for shortest path. Although for each paper p and reviewer r , v p and v r are connected by O ( w ) edges, only 2 of them need to be considered: the ones with the largest cost in both directions v p → v r and v r → v p . Therefore, the shortest path can be computed in O ( n p · n r ) time. Step (6) of Algorithm 1 can be done in O ( n p · n r ) time by maintaining running totals for each reviewer and paper load. And thus Algorithm 1 runs in O ( w · ℓ p · n p 2 · n r ) time.

For (b), let x ∗ be the optimal assignment such that PQuality f ( x ∗ ) = OPT . Consider another assignment x ′ , where x ′ p,r = 1 w ⌊ w · x ∗ p,r ⌋ , ∀ p ∈ P , r ∈ R . Note that x ′ may not be a feasible assignment if ∑ r x ′ p,r ≤ ℓ p for some paper p , but any feasible assignment x ′′ where x ′′ p,r ≥ x ′ p,r for all ( p, r ) will have perturbed quality at least PQuality f ( x ′ ) . Then x ∗ p,r -x ′ p,r ∈ [0 , 1 w ] , and

On the other hand, x ′ corresponds to a feasible flow for the maximum-cost flow problem in Step (5) of Algorithm 1. Therefore, ALG ≥ PQuality f ( x ′ ) . Together with (3), we have ALG ≥ OPT -f ( 1 w ) ∑ p,r S p,r .

B.2 Proof of Theorem 2

Theorem 2. For an input instance ( n p , n r , ℓ p , ℓ r , S ) , where S is blockwise dominant with block identity A ∈ R k × k ≥ 0 and block sizes { p 1 , . . . , p k } , { r 1 , . . . , r k } , assume (i) { r 1 , . . . , r k } are not all equal, (ii) p i · ℓ p ≤ r i · ℓ r ∀ i ∈ { 1 , . . . , k } and (iii) Q · r i ≥ ℓ p ∀ i ∈ { 1 , . . . , k } . Let f be a strictly concave perturbation function and f ′ (0) < Dom( A ) f ′ (1) . PM with f ( x ) as the perturbation function (weakly) dominates PLRA in quality and all randomness metrics. Formally,

(a) ∀ x ∈ PM( Q,f ) , ∀ y ∈ PLRA( Q ) ,

(b) ∀ x ∈ PM( Q,f ) , ∃ y ∈ PLRA( Q ) ,

AvgMaxp( x ) < AvgMaxp( y ) , L2Norm( x ) < L2Norm( y ) , Entropy( x ) > Entropy( y ) .

Proof of Theorem 2: Label the first p 1 papers and r 1 reviewers as group 1 , the next p 2 papers and r 2 reviewers as group 2 , the next p 3 papers and r 3 reviewers as group 3 and so on.

For convenience of language, for an assignment x , define the total Quality of paper p to be ∑ r x p,r S p,r and the total PQuality f of paper p to be ∑ r f ( x p,r ) S p,r .

Define x ∗ to be the assignment such that for each paper p and reviewer r , x ∗ p,r = ℓ p /r i if they are in the same group i and x ∗ p,r = 0 otherwise. Note that x ∗ is feasible, because ℓ p /r i ≤ Q according to assumption (iii), each row of x ∗ sums to ℓ p and each column of x ∗ sums ≤ ℓ r according to assumption (ii). Intuitively, x ∗ is a generalized version of Fig. 1a on blockwise dominant matrices.

Claim B.2.1. For feasible assignment x , Quality( x ∗ ) ≥ Quality( x ) . The equality holds if and only if x assigns paper p to reviewer r with positive probability only when they are in the same group.

Proof of Claim B.2.1: First, Quality( x ∗ ) = ∑ k i =1 p i r i · ℓ p /r i · A i,i = ∑ k i =1 p i · ℓ p · A i,i .

̸

On the other hand, recall that A i,i > A i,j , ∀ j = i . For a paper p in group i , p can only be assigned to ℓ p reviewers, so the total similarity is at most ℓ p · A i,i . Therefore, Quality( x ) ≤ ∑ k i =1 p i · ℓ p · A i,i = Quality( x ∗ ) . The equality holds if and only if each paper in group i is assigned to ℓ p reviewers with A i,i similarity, i.e., ℓ p reviewers in the same group.

As PLRA is maximizing Quality and x ∗ is a feasible solution, we have the following corollary.

Corollary B.2.1. ∀ x ∈ PLRA( Q ) , x assigns paper p to reviewer r with positive probability if and only if they are in the same group.

Next, we consider the performance of PM.

Claim B.2.2. PM( Q,f ) = { x ∗ } .

Proof of Claim B.2.2: First, A i,i ≥ Dom( A ) A i,j , ∀ j = i by the definition of Dom( A ) .

̸

̸

For a paper p in group i , consider the maximum total PQuality f of p . For some assignment x , suppose x assigns p to some reviewer r in group j = i with probability τ . If we adjust x so that the probability τ is instead assigned to reviewers in group i , then the PQuality f will first decrease by f ( τ ) · A i,j ≤ τf ′ (0) A i,j and then increase by at least τf ′ (1) A i,i . As f ′ (0) < Dom( A ) f ′ (1) ,

This shows that the adjustment increases the total PQuality f of p . Therefore, to maximize the total PQuality f of p , p should only be assigned to reviewers in group i . Let p be assigned to the j -th reviewer in group i with probability α j ( ∑ j α j = ℓ p ) , then the total PQuality f of p equals

According to Jensen's inequality, only α j = ℓ p r i , ∀ j maximizes (4). This shows that, for every paper p , the corresponding row of x ∗ is the unique assignment that maximizes the total PQuality f of p . And thus x ∗ is the unique solution that maximizes the overall PQuality f .

Now we are ready to proceed to prove Theorem 2. Note that Claim B.2.2 shows that the only possible solution of PM is x ∗ . For clearer presentation, let x (PM) = x ∗ .

For Quality , Claim B.2.1 implies ∀ x (PLRA) ∈ PLRA( Q ) , Quality( x (PM) ) ≥ Quality( x (PLRA) ) .

For Maxprob , compute that Maxprob( x (PM) ) = max i { ℓ p /r i } . On the other hand, Corollary B.2.1 implies that ∀ x (PLRA) ∈ PLRA( Q ) , Maxprob( x (PLRA) ) ≥ max i { ℓ p /r i } = Maxprob( x (PM) ) .

For Support , compute that Support( x (PM) ) = ∑ k i =1 p i r i . On the other hand, Corollary B.2.1 implies that ∀ x (PLRA) ∈ PLRA( Q ) , Support( x (PLRA) ) ≤ ∑ k i =1 p i r i = Support( x (PM) ) .

For the other metrics, AvgMaxp , Entropy and L2Norm , consider a paper p in group i . By Corollary B.2.1, we know that PLRA only assigns p to reviewers in group i . Let it be assigned to the j -th reviewer in group i with probability α j ( ∑ j α j = ℓ p ) . Note that max j { α j } is minimized by α j = ℓ p r i , ∀ j , which is the corresponding row in x (PM) for each p . Meanwhile, according to Jensen's inequality, ∑ j α 2 j is minimized by α j = ℓ p r i , ∀ j and ∑ j α j ln(1 /α j ) is maximized by α j = ℓ p r i , ∀ j . So ∀ x (PLRA) ∈ PLRA( Q ) , AvgMaxp( x (PM) ) ≤ AvgMaxp( x (PLRA) ) , L2Norm( x (PM) ) ≤ L2Norm( x (PLRA) ) and Entropy( x (PM) ) ≥ Entropy( x (PLRA) ) . Moreover, as { r 1 , . . . , r k } are not all the same, for some group i , ℓ p r i < max j { ℓ p r j } ≤ Q . For this group, α j = ℓ p r i , ∀ j is not the only solution. Thus, ∃ x (PLRA) ∈ PLRA( Q ) , AvgMaxp( x (PM) ) < AvgMaxp( x (PLRA) ) , L2Norm( x (PM) ) < L2Norm( x (PLRA) ) and Entropy( x (PM) ) > Entropy( x (PLRA) ) .

This concludes the proof of Theorem 2

B.3 Proof of Theorem 3

Theorem 3. For an input instance ( n p , n r , ℓ p , ℓ r , S ) , where each entry of S is i.i.d . sampled from { v 1 , . . . , v k } (0 < v 1 < · · · < v k ) uniformly, assume (i) k ≤ 1 c · n p , (ii) ℓ r ≥ c · ln( n r ) , (iii) 2 · n p · ℓ p ≤ n r · ℓ r and (iv) Q · ( n r -1) ≥ ℓ p . Let f be a strictly concave perturbation function and f ′ (0) < v i v i -1 f ′ (1) , ∀ i ∈ { 2 , . . . , k } . With probability 1 -e -Ω( c ) , PM with f ( x ) as the perturbation function (weakly) dominates PLRA in quality and all randomness metrics. Formally,

(a) ∀ x ∈ PM( Q,f ) , ∀ y ∈ PLRA( Q ) ,

Quality( x ) ≥ Quality( y ) , Maxprob( x ) ≤ Maxprob( y ) , AvgMaxp( x ) ≤ AvgMaxp( y ) , L2Norm( x ) ≤ L2Norm( y ) , Entropy( x ) ≥ Entropy( y ) , Support( x ) ≥ Support( y ) .

(b) ∀ x ∈ PM( Q,f ) , ∃ y ∈ PLRA( Q ) ,

Support( x ) > Support( y ) , L2Norm( x ) < L2Norm( y ) , Entropy( x ) > Entropy( y ) .

Proof of Theorem 3: To prove this statement, first consider the following 2 algorithms.

Algorithm 4 : Greedy

For each paper p , let the reviewers sorted by decreasing similarity with p be { r 1 , r 2 , . . . , r n r } . Greedily assign p to the first few reviewers as follows: assign p to r 1 , r 2 , · · · , r ⌊ ℓ p /Q ⌋ with probability Q , and assign p to r ⌊ ℓ p /Q ⌋ +1 with probability ℓ p -Q ⌊ ℓ p /Q ⌋ .

Algorithm 5 : Balanced Greedy

For each paper p , suppose ∀ i ∈ { 1 , 2 , . . . , k } the set of reviewers with similarity v i is R p,i . Initialize remaining paper requirement ℓ remain = ℓ p . For i from k to 1 :

If ℓ remain ≥ Q · | R p,i | , assign p to each reviewer in R p,i with probability Q .

Otherwise uniformly assign p to each reviewer in R p,i with probability ℓ remain / | R p,i | .

Update ℓ remain to the current remaining paper requirement max(0 , ℓ remain -Q | R p,i | ) .

Note that Balanced Greedy is almost the same algorithm as Greedy, except that Balanced Greedy groups reviewers with the same similarity v i with p together as R p,i and always assigns the same probability to them, while Greedy treats each reviewer individually.

These two Greedies do not always produce feasible assignments because they both consider each paper p individually and do not take the constraint that each reviewer is assigned at most ℓ r papers into account. Nevertheless, we will show that with high probability, they are feasible, and if this is the case, we can relate them to PM & PLRA and prove the Theorem 3.

At a high level, we will prove the Theorem 3 with the following steps.

Greedy and Balanced Greedy are feasible with probability 1 -e -Ω( c ) .

When Greedy and Balanced Greedy are feasible:

Greedy produces a possible solution from PLRA.

PM does exactly the same with Balanced Greedy.

Balanced Greedy dominates Greedy with probability 1 -e -Ω( c ) .

We will formalize and prove the above 3 steps below.

Claim B.3.1. Greedy and Balanced Greedy are feasible with probability 1 -e -Ω( c ) .

Proof of Claim B.3.1: We use Bernstein's inequality: Suppose x 1 , x 2 , . . . , x n are i.i.d . from a distribution with mean µ , bounded support [ a, b ] , with variance σ 2 . Then

In Greedy and Balanced Greedy, the output x is a n p × n r matrix of random variables, where variables from different rows (papers) are independent as different papers are considered individually. These algorithms are feasible if and only if for each reviewer r , ∑ p x p,r ≤ ℓ r . Note that as the similarities are i.i.d. random, the reviewers are symmetric. Using Union Bound across reviewers,

Moreover, the papers are also symmetric. So ∑ p x p,r 1 is then a sum of n p i.i.d . random variables. Let their distribution be D . We will bound the probability using Bernstein's inequality. To do this, consider the properties of the distribution D . For both Greedy and Balanced Greedy,

The expectation µ of D should be ℓ p /n r .

Also, D is supported on [0 , Q ] ⊆ [0 , 1] .

Therefore, σ 2 = E x ∼ D [ x 2 ] -µ 2 ≤ E x ∼ D [ x 2 ] ≤ E x ∼ D [ x ] = ℓ p /n r .

Also, by assumption (iii), 2 · n p · ℓ p ≤ n r · ℓ r , we then know ℓ r /n p ≥ 2 · ℓ p /n r . So that

Letting t = ℓ r 2 n p , we will use Bernstein's inequality on this formula. According to Bernstein's inequality,

Then, by assumption (ii), ℓ r ≥ c · ln( n r ) , and we will see ∀ ALG ∈ { Greedy , Balanced Greedy } ,

This concludes the proof of Claim B.3.1

For convenience of languague, for an assignment x , define the total Quality of paper p to be ∑ r x p,r S p,r and the total PQuality f of paper p to be ∑ r f ( x p,r ) S p,r .

Let x ( b ) = Balanced Greedy( Q ) , x ( g ) = Greedy( Q ) . For paper p , both Balanced Greedy and Greedy maximize the total Quality of p , so Quality( x ( b ) ) = Quality( x ( g ) ) . Moreover, this property of the Greedies also gives us the following Claim B.3.2.

Claim B.3.2. For any feasible assignment x , Quality( x ( b ) ) ≥ Quality( x ) . The equality holds if and only if for each paper p , x maximizes the total Quality of p .

And as PLRA is maximizing Quality , we further get the following Corollary B.3.1.

Corollary B.3.1. If x ( b ) and x ( g ) are feasible, then x ( b ) , x ( g ) ∈ PLRA( Q ) and ∀ x ∈ PLRA( Q ) , x maximizes the total Quality of p .

Next, we consider the performance of PM.

Claim B.3.3. For each paper p , x ( b ) uniquely maximizes the total PQuality f of p .

Proof of Claim B.3.3: For each paper p , recall that ∀ i ∈ { 1 , 2 , . . . , k } , the set of reviewers with similarity v i to p is R p,i . By the execution of Balanced Greedy, x ( b ) assigns p to every reviewer in the i -th set, R p,i , with the same probability. Let this probability be A p,i . Then, there exists a t ∈ { 1 , 2 , . . . , k } , such that A p,t +1 = · · · = A p,k = Q , A p,t < Q and A p,t -1 = · · · = A p, 1 = 0 .

Suppose that for an assignment x , there are both a reviewer r high ∈ R p,i such that x p,r high < Q , and another reviewer r low ∈ R p,j ( j < i ) such that x p,r low > 0 . Let δ = min { Q -x p,r high , x p,r low } . Consider adjusting x by decreasing x p,r low by δ and increasing x p,r high by δ . The total PQuality f of

p will first decrease by at most δf ′ (0) v j and then increase by at least δf ′ (1) v i . The net increase in the total PQuality f of p will be

This shows that the adjustment increases the total PQuality f of p , so x does not maximize it. Therefore, for an assignment x to maximize the total PQuality f of p , x must assign p to every reviewer in R p,t +1 , . . . , R p,k with probability Q , and assign p to every reviewer in R p,t -1 , . . . , R p, 1 with probability 0 . It remains to consider the assignment to group R p,t .

Write the total PQuality f of p in x as

For fixed p , the first summation is constant. To maximize the second summation, according the Jensen's inequality, x p,r , ∀ r ∈ R p,t must be the same, which is exactly x ( b ) .

Claim B.3.3 gives us the following Corollary B.3.2.

Now we are ready to proceed to prove Theorem 3. We first prove Theorem 3 (a).

For the randomness metrics, consider each paper p and again let the set of reviewers with similarity v i to p be R p,i . Like in the proof of Claim B.3.3, by the execution of Balanced Greedy, x ( b ) assigns p to every reviewer in R p,i with the same probability A p,i , and there exists a t ∈ { 1 , 2 , . . . , k } , such that A p,t +1 = · · · = A p,k = Q , A p,t < Q and A p,t -1 = · · · = A p, 1 = 0 .

By Corollary B.3.1, ∀ x ∈ PLRA( Q ) , x must assign p to all reviewers in R p,t +1 , . . . , R p,k with probability Q and assign p to all reviewers in R p, 1 , . . . , R p,t -1 with probability 0 . Among all such assignments, with an argument using Jensen's inequality or simple categorical discussion, we will see that x = x ( b ) maximizes ∑ r x p,r ln(1 /x p,r ) , ∑ r I [ x p,r > 0] and minimizes max r { x p,r } , ∑ r x 2 p,r .

This shows that Theorem 3 (a) holds.

To show Theorem 3 (b), we will first prove the following Claim B.3.4.

Proof of Claim B.3.4: Recall in Greedy, for each paper p , the sorted reviewer list by decreasing similarity with p is { r 1 , r 2 , . . . , r n r } . By assumption (iv), Q · ( n r -1) ≥ ℓ p , so ⌈ ℓ p /Q ⌉ +1 ≤ n r . Suppose for some paper p and i ∈ { 1 , . . . , k } , S p,r ⌈ ℓ p /Q ⌉ = S p,r ⌈ ℓ p /Q ⌉ +1 = v i . Then, as by the execution of Greedy, x ( g ) p,r ⌈ ℓ p /Q ⌉ > 0 and x ( g ) p,r ⌈ ℓ p /Q ⌉ +1 = 0 . Let { r left , . . . , r right } be the set of reviewers with similarity v i to p , where left ≤ ⌈ ℓ p /Q ⌉ < ⌈ ℓ p /Q ⌉ +1 ≤ right . Then

Note that x ( b ) p,r j = x ( g ) p,r j for any j ≤ left -1 and any j ≥ right + 1 . For left ≤ j ≤ right , x ( b ) p,r j are all equal, but as x ( g ) p,r ⌈ ℓ p /Q ⌉ +1 = 0 , x ( g ) p,r j are not all equal. So Support( x ( b ) ) > Support( x ( g ) ) , and by Jensen's inequality, L2Norm( x ( b ) ) < L2Norm( x ( g ) ) , Entropy( x ( b ) ) > Entropy( x ( g ) ) .

Therefore, it remains to show that with probability 1 -e -Ω( c ) , for some paper p , S p,r ⌈ ℓ p /Q ⌉ = S p,r ⌈ ℓ p /Q ⌉ +1 . For a fixed p , denote the event that S p,r ⌈ ℓ p /Q ⌉ = S p,r ⌈ ℓ p /Q ⌉ +1 as E p . Recall that entries in S are i.i.d. and uniformly chosen from { v 1 , . . . , v k } . Consider fixing S p, 1 , . . . , S p,n r -1 , and let the ⌈ ℓ p /Q ⌉ -th largest number in this set be α . Then Pr[ S p,n r = α ] ≥ 1 k . As S p,n r = α implies E p , Pr[ E p = 1] ≥ 1 k . Therefore, according to assumption (i), k ≤ 1 c · n p , we know that

This concludes the proof of Claim B.3.4.

Theorem 3 (b) is implied by Claim B.3.1, Corollary B.3.1, Corollary B.3.2 and Claim B.3.4.

C Discussions

C.1 Directly Optimizing Specific Randomness Metrics with PM

While PM provides one way to introduce randomness into the paper assignment, one natural alternative approach is to simply maximize one specific randomness metric, subject to a constraint on the minimum solution quality. In this section, we show that a slight modification to PM is general enough to capture such approaches.

Specifically, suppose we want to maximize a concave randomness metric RM : [0 , 1] n p × n r → R over the set of feasible assignments S feasible = { x ∈ [0 , 1] n p × n r | ∑ r x p,r = ℓ p , ∀ p and ∑ p x p,r ≤ ℓ r , ∀ r } subject to a minimum requirement of the solution quality Quality( x ) ≥ τ · Quality OPT , where Quality OPT = max x ∈ S feasible Quality( x ) and τ ∈ [0 , 1] . The problem can be formulated as

We then consider a slight generalization of PM that allows different perturbation functions for each reviewer-paper pair. That is, we change the definition of PQuality f (the objective of PM) in Definition 4.2 to

Then we have the following result:

Theorem 4. If f p,r ( x ) is (i) x + λ S p,r I [ x > 0] , (ii) x -λ S p,r x ln( x ) , or (iii) x -λ S p,r x 2 , then PM achieves the optimal trade-offs between Quality( x ) and (i) Support( x ) , (ii) Entropy( x ) , or (iii) L2Norm 2 ( x ) respectively with different values of λ ∈ [0 , + ∞ ) .

In fact, the proof of Theorem 4 also shows that PM-Q can be viewed as a algorithm that achieves the optimal trade-off between Quality( x ) and another randomness metric ( ∑ p,r S p,r x 2 p,r ) , a similarityweighted version of squared L2 norm of x . Next, we will proceed to prove Theorem 4.

Proof of Theorem 4: When τ ∈ [0 , 1] , the maximization problem (5) always has at least one feasible solution. Let the optimal value of (5) be RM ∗ ( τ ) . RM ∗ ( τ ) describes the optimal trade-off between Quality( x ) and RM( x ) . We can show the following property of RM ∗ ( τ ) .

Lemma C.1. RM ∗ ( τ ) is a concave and non-increasing function of τ ∈ [0 , 1] .

Proof of Lemma C.1: Let x 1 be the optimal solution of (5) when τ = τ 1 and x 2 be the optimal solution of (5) when τ = τ 2 where 0 ≤ τ 1 < τ 2 ≤ 1 . Then, we have Quality( x 2 ) ≥ τ 2 · Quality OPT > τ 1 · Quality OPT and x 2 is a feasible solution of of (5) when τ = τ 1 . Therefore, RM( x 1 ) ≥ RM( x 2 ) , i.e., RM ∗ ( τ 1 ) ≥ RM ∗ ( τ 2 ) . This shows that RM ∗ ( τ ) is a non-increasing function of τ .

Moreover, let x m = 1 2 ( x 1 + x 2 ) . Then, Quality( x m ) ≥ 1 2 ( τ 1 + τ 2 ) · Quality OPT and x m ∈ S feasible . Thus x m is a feasible solution of of (5) when τ = 1 2 ( τ 1 + τ 2 ) . Since RM is a concave function,

RM ∗ ( 1 2 ( τ 1 + τ 2 )) ≥ RM( x m ) ≥ 1 2 (RM( x 1 ) + RM( x 2 )) = 1 2 (RM ∗ ( τ 1 ) + RM ∗ ( τ 2 )) . This shows that RM ∗ ( τ ) is a concave function of τ .

Now, for λ ≥ 0 , consider another optimization program as follows:

Let the solution of (6) be x ∗ λ . Lemma C.1 gives us the following corollary.

Corollary C.1.1.

Plugging in different randomness metrics RM( x ) concludes the proof of Theorem 4.

C.2 Incorporating Various Constraints in PM

In some of the currently-deployed conference review systems, there are various additional constraints on the assignment of papers to reviewers. We will discuss in this section how to incorporate some of these constraints in PM. Specifically, we consider the following constraints from [20]:

Seniority: Each paper is assigned to ≥ 1 senior reviewer.

Geographic diversity: No 2 reviewers assigned to the same paper belong to the same region.

Seniority. Let the set of senior reviewers be S senior . We can incorporate the seniority constraint in PM by modifying PM to the following optimization program.

With a slightly different sampling algorithm, we can show that the obtained randomized assignment can be realized by a distribution of deterministic assignments that satisfies the seniority constraint [22].

Geographic diversity. Let the set of regions be S region . We can incorporate the geographic diversity constraint in PM by modifying PM to the following optimization program.

The obtained assignment can also be realized by a distribution of deterministic assignments that satisfies the geographic diversity constraint with a slightly different sampling algorithm [22].

Apart from the constraints mentioned above, there are also some other common constraints that cannot be easily incorporated in PM [20]. We leave incorporating them for future work.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on randomized paper assignments in the reviewer-paper matching problem. First, the authors state the motivation and emphasize the importance of randomness in paper assignments. Then, they point out the weakness of the existing method, PLRA, and propose additional metrics for evaluating the performance of randomized paper assignments. After that, the authors design the PM-E and PM-Q algorithms with different perturbation functions to solve the proposed problem and provide theoretical analysis to prove the effectiveness of the proposed algorithms. Finally, various experiments have been conducted, showing that PM-E and PM-Q perform well on real-world datasets. In conclusion, this paper proposes practical and universal algorithms, PM-E and PM-Q, with the aim of mitigating malicious behaviors, facilitating assignment evaluation, and increasing reviewer diversity and anonymity. However, the theoretical analysis for the correctness of the approximate algorithm is inadequate.\n\nSTRENGTHS:\n1. This paper is well-written and easy to understand.\n2. Related works are described in detail and the limitations of existing methods are well illustrated by a simple example.\n3. The theoretical analyses regarding how the proposed PM algorithm outperforms PLRA are sufficient.\n4. Various experiments show that the proposed algorithm has better performance than existing randomzied paper assignment methods.\n\nWEAKNESSES:\n1. Algorithm 1 prposes a network-flow-based approximation of PM and use experiments to show its pratical performance, but the theoretical analysis of how this algorithm approximate solves PM problem and approximation ratio is missing.\n2. The motivations listed in Introduction mainly talk about randomness in paper assignments, but randomized paper assignment problem has been proposed in PLRA.\n\nQUESTIONS:\n1. Why is the additional metric valid? Could the authors provide some analysis or examples to briefly explain it?\n2. Figure 1 presents the ideal assignment, but it assigns more papers to each reviewer. Therefore, is it possible that in a scenario where the quality provided by PLRA is similar to PM-E and PM-Q, PLRA assigns fewer articles to each reviewer resulting in a lower workload for each reviewer?\n\nLIMITATIONS:\nThis paper acknowledges the limitation that deploying a randomized assignment may have a negative impact on assignment quality compared to deterministic assignments. Additionally, the authors propose a reasonable explanation that their work allows conferences to choose the level of randomization to address this limitation.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
In this paper, the authors propose a new algorithm to improve randomness in peer-review paper assignment without sacrificing assignment quality. The authors first identify a set of good randomness properties in paper assignment besides the max probability used in the PLRA. The authors propose a simple and practical variation to PLRA by adding a concave perturbation function. Empirical evaluation on two real-world datasets demonstrates superior performance of the proposed method to PLRA.

STRENGTHS:
1. The paper is well written with clear motivation and justification of the method. The identification of good randomness properties could motivate future research on random paper assignments.
2. The proposed algorithm is very practical and easy to implement as convex optimization which can be solved efficiently with commercial software.
3. The authors demonstrate the superiority of the proposed method both theoretically and empirically on two real-world peer-review datasets.

WEAKNESSES:
1. Though the authors provide comprehensive comparison to PLRA, it would be interesting to see comparison of the proposed method to other baselines, even the deterministic ones. Also, it would be interesting to provide “randomness” metrics from samples of assignments.
2. The authors motivate the requirement of randomness from 4 perspectives. It would be interesting to design more direct metrics or evaluation in terms of these 4 perspectives.
3. It would be better to include experiment and discussion on hyper-parameter sensitivity. As both PM-Q and PM-E contain the same number of parameters as in PLRA.

QUESTIONS:
N/A

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an improvement on PLRA, an important (used in major reviewing deployments like AAAI and OpenReview) algorithm for randomizing reviewer assignments while preserving a prescribed fraction of assignment quality, by finding randomized assignments that increase additional randomization metrics of interest beyond bounding the probability a particular reviewer is assigned to a particular paper. These additional metrics help find solutions of comparable quality (as defined by PLRA) that address additional concerns such as increasing average randomness (intended to further guard against malicious behavior) and the support size (useful for better off-policy evaluation). The work is soundly evaluated theoretically and empirically.\n\nSTRENGTHS:\nThe paper makes a significant contribution to reviewer assignment algorithms, an important topic relevant to the NeurIPS community. The proposed algorithm substantially improves on the previous algorithm with little to no apparent downside, and can be readily deployed. The work makes an original contribution and is presented with very high clarity.\n\nWEAKNESSES:\nThe paper is very strong, but some minor weaknesses remain. The work relaxes the L2 norm constraint employed by PLRA to a soft constraint, but it is unclear how this relaxation might negatively harm results. The proposed algorithm appears novel in the context of reviewer assignment, but novelty in the context of related problems is less clear - it would be better to clarify the extent to which the technical contribution leverages known techniques (e.g., max flow) and how different this application is from related problems that have employed similar techniques in the past. The paper empirically tests on only 2 datasets, where it would seem easy to evaluate on additional datasets. See also my questions in the next section.\n\nQUESTIONS:\n- The proposed approach improves on PLRA but how does it compare to [20,29,30]?\n- How much does relaxing the L2 norm to a soft constraint instead of the hard constraint used by PLRA negatively impact results?\n- How were AAMAS and ICLR 2018 chosen, and why were additional datasets not also experimented with? Would the empirical results generalize?\n- How can conference chairs prioritize certain proposed metrics over others? Is there a relative tradeoff?\n- How/why was 95% picked for Table 1, versus also providing results for other percentages 80-100%?\n\nLIMITATIONS:\nThe limitations could include more details on the limitations of the empirical experiments such as potential lack of generality from testing on only 2 datasets.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work is concerned with randomness in paper matching for peer review. It provides normative motivations for the value of randomness in this process and proposes a family of generalizations of the Probability Limited Randomized Assignment (PLRA) paper matching framework of Jecmen et al. This generalization, which the authors term Perturbed Maximization, entails replacing the linear objective in the PLRA linear programming problem with a concave objective which increasingly discounts paper-reviewer matchings with higher probabilities. The authors prove comparisons between the two methods under two distributions of paper-reviewer similarity scores. Experimentally, the authors consider two choices of discounting (perturbation) functions, demonstrate that solving this convex problem is tractable in practice, and present the effect of deploying their method for two past conferences, according to multiple measures of randomness.\n\nSTRENGTHS:\nThis work proposes and studies a natural generalization of the PLRA paper matching framework. The generalization and analysis are particularly suited to practical use, in the generalization can be arbitrarily incremental, and the PLRA framework has been used in practice and serves as a benchmark for the experiments.\n\nFrom a theoretical perspective, it is interesting to identify combinatorial optimization problems/applications for which randomness is a desirable property, and to study how randomness this trades off against optimality.\n\nThe experimental results seem to demonstrate that, at least for some paper matching instances, the framework could be used to generate matchings with comparable quality in a much more randomized way.\n\nWEAKNESSES:\nTheoretical results: I have a difficult time understanding the settings and claims of the theoretical analysis provided, and I am suspicious of the implications of the analysis to understanding PM as compared to PLRA. The quantity c in Theorem 3 is unspecified, and the claims seem to suggest that the models permit either the same or somehow degenerate solution sets for the two algorithms. In any case, more explanation of these claims seems warranted.\n\nRandomness metrics: the motivation for the proposed randomness metrics for the distribution x in Section 3 is very thin. Some references for these being useful measures for a fractional 'matching' in this application or elsewhere would be welcome. But there seems to be a deeper mismatch between these measures and the reasons randomness is desirable outlined in Section 1. Properly, (and as observed in the discussion of entropy), x is not a distribution; instead the fractional assignment x together with a dependent rounding scheme together induce a distribution over reviewer-paper assignments.  The randomness of this distribution seems arguably more relevant to most of the randomness motivations provided in Section 1; for instance, depending on the dependent rounding scheme, a fractional assignment x with Q=0.1 could correspond to sampling one of only 10 final assignments, allowing easy reviewer de-anonymization. I think that calculating (or estimating) these randomness metrics for the assignment distribution would lead to a superior analysis.\n\nQUESTIONS:\nRandomness metrics:\n\tCan you remark on which metrics in Section 3 would be natural choices for any of the randomness motivations in Section 1, and whether you agree that in any cases the corresponding metrics for the induced assignment distribution would be more relevant? \t\t \t\n\tAlternatively, do you think there is good motivation for restricting your randomness metrics to ones that are linear over the fractional assignments of individual papers (as all except L2 are)?\n\tDo the authors of [22] mention or consider any metrics other than Maxprob?\n\nTheoretical results:\n\tWhat is the scalar c in the statement of Theorem 3? Is it an unspecified constant, or does the statement hold for every c?\n\tIf part (a) of Theorems 2 and 3 holds for all pairs of points in the solution set, the claimed Quality inequality seems to suggest that the solution set of PM is contained in the solution set for PLRA. This in turn raises two questions: first, given that the PM objective is different, wouldn't this suggest that these theoretical settings are in a sense too degenerate (have too many equally optimal solutions/too few nearby suboptimal ones) to distinguish between the two methods? Second, do the remaining claimed inequalities suggest that PM identifies a unique point in PLRA(Q) which simultaneously minimizes/maximizes those randomness measures within the set PLRA(Q)?\n\nOn line 333, how does Figure 4 show that PM sacrifices Maxprob relative to PLRA?\n\nLIMITATIONS:\nThe choice of perturbation function seem ad hoc, and it is not obvious why the authors decided to apply the perturbation function to the full range [0,1] instead of to the range [0,Q]. For the perturbation functions studied here this does not matter, but in general it could. There are three places in the narrative where the authors present a range of options: (i) motivations for randomness in paper matching, (ii) measures of randomness, and (ii) choices of perturbation function. It would have been interesting to see some discussion of how choices of (i) might motivate different choices of (ii), particularly with regard to motivations 2 and 4, and some discussion of how different choices of dependent rounding scheme given the fractional assignment affect these objectives. But it would have been particularly nice to see some motivation of how choices of (ii) inform choices of (iii). For instance, are there informal or heuristic arguments that would suggest that one form of perturbation function would be particularly suited to maximizing the entropy objective?\n\nThe theoretical analysis is a little unsettling. Either the theoretical similarity models considered seem inapt to the purpose of comparing the two methods, or the theoretical claims could use significantly more contextualization. \n\nCertain aspects of the experimental section could be clearer. In particular, it would be helpful to include a slightly more explicit description of the hyperparameters that are being fit, and the order in which they are being fixed. In my opinion this omission makes the presentation and interpretation of Figure 2 particularly misleading, since the form it takes and the insights offered are directly determined by the value assigned to the hyperparameter delta, which is not mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}