Group Fairness in Peer Review
Authors: Haris Aziz
Evi Micha
Nisarg Shah
Venue: NeurIPS 2023 spotlight (2023)
Keywords: peer review; group fairness; core; stable
TL;DR: We define group fairness in a peer review assignment problem using the concept of core stability and show that a review assignment in the core can be efficiently found when each submission is single-authored.

Abstract: Large conferences such as NeurIPS and AAAI serve as crossroads  of various AI fields, since they attract submissions from a vast number of communities. However, in some cases, this has resulted in a poor reviewing experience for some communities, whose submissions get assigned to less qualified reviewers outside of their communities. An often-advocated solution is to break up any such large conference into smaller conferences, but this can lead to isolation of communities and harm interdisciplinary research. We tackle this challenge by introducing a  notion of group fairness, called the core, which requires that every possible community (subset of researchers) to be treated in a way that prevents them from unilaterally benefiting by  withdrawing from a large conference. 

We study a simple peer review model, prove that it always admits a reviewing assignment in the core, and design an efficient algorithm to find one such assignment. 
We use real data from CVPR and ICLR conferences to compare our algorithm to existing reviewing assignment algorithms on a number of metrics.

Group Fairness in Peer Review
Haris Aziz UNSW Sydney haris. wwlisaazzzuunss.ee. edu. au
Evi Micha University of Toronto emi cha@cs. www.stt.ooooooo.o edu
Nisarg Shah
University Toronto
nisargecs sisaag@cs.torrntoooooo toronto.
Abstract
Large conferences such as NeurlPS and AAAI serve as crossroads of various Al fields, since they attract submissions from a vast number of communities. However, in some cases, this has resulted in poor reviewing experience for some communities, whose submissions get assigned to less qualified reviewers outside of their communities. An often-advocated solution is to break up any such large conference into smaller conferences, but this can lead to isolation communities and harm interdisciplinary research. We tackle this challenge by introducing a notion of group fairness, called the core, which requires that every possible community (subset of researchers) be treated in a way that prevents them from unilaterally benefiting by withdrawing from a large conference.
We study simple peer review model, prove that it always admits a reviewing as- signment the core, and design an efficient algorithm find one such assignment. We use real data from CVPR and ICLR conferences to compare our algorithm to existing reviewing assignment algorithms on a number of metrics.
1 Introduction
Due to their large scale, conferences like NeurIPS and AAAI use an automated procedure to assign submitted papers to reviewers. Popular such systems include the Toronto Paper Matching System [1], Microsoft CMT', and OpenReview?. The authors submitting their works are often very interested in receiving meaningful and helpful feedback from their peers [2-4]. Thus, their overall experience with the conference heavily depends on the quality of reviews that their submissions receive.
The typical procedure of assigning papers to reviewers is follows. First, for each paper-reviewer pair, a similarity score calculated based on various parameters such as the subject area of the paper and the reviewer, the bids placed by the reviewer, etc. [1. 5-8J. Then, assignment is calculated through an optimization problem, where the usual objectives are to maximize either the utilitarian social welfare, which is the total similarity score of all matched paper-reviewer pairs, or the egalitarian social welfare, which is the least total score of reviewers assigned to any submission. Relevant constraints imposed to ensure that each submission receives an appropriate number of reviews, reviewer workloads are respected, and any conflicts interest are avoided.
Peng et al. [9] recently mentioned that major problem with the prestigious mega conferences that they constitute the main venues for several communities, and as a result, some cases, people are asked to review submissions that are beyond their main areas of work. They claim that reasonable solution to move to de-centralized publication process by creating more specialized conferences appropriate for different communities. While specialized conferences definitely have their advantages, the maintenance of large conferences that attract multiple communities is also crucial for the emergence of interdisciplinary ideas that can be reviewed by diverse subject experts. Therefore, it is important to ensure that no group has an incentive to break off due to a feeling of being mistreated by the reviewing procedure of large conference. In this work, we ask whether
'nttps://cmt3, www.etrc.eeeerc..cmmmmamo microsoft. com/
"nttpa://githan com/openreview/ ctmp//ppeneeviieeepppeeeeeeeeeee wttp:///pppeeeeeeeeeeeeeeee matcher
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

is possible to modify the existing reviewing processes to resolve this issue by treating the various communities satisfactorily. We clarify that the goal is not to build roadblocks to the creation of specialized conferences, but rather to mitigate the harm imposed on communities in large conferences.
To answer this, we look toward the literature on algorithmic fairness. Specifically, we adapt the group fairness notion known as the core [10]: to the best of our knowledge, we are the first to introduce it the peer review setting. For a reviewing assignment be in the core, must ensure that no community (subset of researchers) can "deviate" by setting up its own conference in which (a) no author reviews her own submission, (b) each submission from within the community is reviewed by just as many reviewers as before, but now from within the community, (c) each reviewer reviews no more papers than before, and (d) the submissions are assigned to better reviewers, making the community happier. Intuitively, this is notion of group faimess because it ensure that the treatment provided to every group of participants meets their "entitlement" (which defined by what the group can achieve on its own). It is also notion of stability (often also known as core stability) because it provides no incentives for any community to break off and get isolated by setting up their own conference instead. Note that this definition provides faimesss to every possible community, and not only to predefined groups, as is the case for fairness definitions such as demographic parity and equalized odds that are popular in the machine learning literature [11]. In particular, it ensures fair treatment to emerging interdisciplinary communities even before they become visible.
1.1 Our Contribution
We consider a simple peer review model in which each agent submits (as the sole author) a number of papers to conference and also serves a potential reviewer. A reviewing assignment is valid if each paper is reviewed by kp reviewers, each reviewer reviews no more than ka papers, and no agent reviews her own submissions. To ensure that valid assignment always exists, we assume that the maximum number of papers that each agent is allowed to submit is at most [ka/kip]-
In Section 3, we present an efficient algorithm that always returns a valid assignment in the core under minor conditions on the preferences of the authors. Specifically, our algorithm takes as input only the preference ranking of each author over individual potential reviewers for each of her submissions. Then, produces an assignment that we prove to be in the core for any manner in which the agent's ' subnission-specifie preferences over individual reviewers may be extended to preferences over set of ko reviewers assigned to each submission, aggregated over submissions, subject to two mild conditions being satisfied.
In Section 4, we conduct experiments with real data from CVPR and ICLR conferences, and evaluate pay - in lost utilitarian and egalitarian welfare - in order to satisfy the core and prevent communities from having an incentive to deviate. We also observe that reviewer assignment methods currently used in practice generate such adverse incentives quite often.
the price that our algorithm must
1.2 Related Work
As we mentioned above, usually the first step of a review assignment procedure is to calculate similarity score for each pair of submission and reviewer which aims to capture the expertise of the reviewer for this submission. The problem of identifying the similarity scores has been extensively studied in the literature [1, 5-8, 12). In this work, we assume that the similarity scores are given as an input to algorithm after they have been calculated from procedure that is considered as a black box. Importantly, our algorithm does not need the exact values of the similarity scores, but it only requires ranking of the reviewers for each paper, indicating their relative expertise for this paper.
Given, the similarities scores various methods have been proposed for finding reviewing assignment. The most famous algorithm is the Toronto Paper Matching System [1] which is a very broadly applied method and focuses on maximizing the utilitarian welfare, i.e. the sum of the similarities across all assigned reviewers and all papers. This approach has been adopted by other popular conference management systems such as EasyChair' and HoICRP [13]. While this approach optimizes the total welfare, it is possible to discriminate against some papers. Therefore, other methods have focused on finding reviewing assignments that are (also) fair across all papers.
hnctpe://easychhir..n 'nteps//hhoccpp.com
2

---

O'Dell et al. [14] suggest a method where the goal is to maximize the total score that a paper gets, while Stelmakh et al. [13] generalized this method by maximizing the minimum paper score, then maximizing the next smallest paper score, etc. Hartvigsen et al. [15] ensure fairness by requiring that each paper assigned at least one qualified reviewer. Kobren al. [16] proposed two algorithms that maximize that total utilitarian under the constraint that each paper should receive a score that exceeds a particular threshold. Payan and Zick [17] used the idea of envy-freeness [18] from the fair division literature to ensure fairnsss over the submissions. Moreover, some other works have focused on being fair over the reviewers rather than the papers [19, 20D. The core property that we consider faimess requirement over groups of authors. The reader can find
in this work can be viewed more details about the challenges of the peer review problem in the recent survey of Shah [21].
In our model, the review assignment problem is related to exchange problems with endowments [22], since authors can viewed as being endowed by their own papers which they wish to exchange with other authors that also serve as reviewers. For the basic exchange problem of housing reallocation, Shapley and Scarf [22] showed that an algorithm called Top-Trading- Cycle (TTC) finds an allocation which is in the core. The first part of our algorithm uses a variation of TTC where the agents (authors) are incorporated with multiple items (submissions), and constraints related to how many items each agent can get and to how many agents one item should be assigned should be satisfied. In contrast to classical exchange problem with endowments, our model has a distinctive requirement that agents/authors need to give away all their items/papers as the papers need to be reviewed by the agent who gets the paper. As we further explain in Section 3, this difference is crucial and requires further action from our algorithm than simply executing this variation of TTC. Various variations of TTC have been considered in the literature, tailored for different variations of the basic problem, but to the best of our knowledge, none of them can be directly applied in our model. To give an example, Suzuki et al. [23] consider the case that there are multiple copies of the same object and there are some quotas that should be satisfied, but they assume that each agent gets just one object while here each paper is assigned to multiple distinct reviewers.
2 Model
.4... There is a set of agents N = =[n]. Each agent i submits a set of papers P, .Pr......PPI.... } for review by her peers, where m, E N. and is available to review the submissions of her peers. We refer to Pi, as the f-th submission of agent i; when considering the special case of each agent i having single submission, we will drop and simply write Pt. Let P = UiexP, P, be the set of all submissions and m = Σien m; be the total number of submissions.
For q E N. define [4] A {1,.....):
Assignment. Our goal is to produce a (reviewing) assignment R N x P + f0, 1}. where R(i, 1 if agent N is assigned to review submission P. With slight abuse of notation, let R2 = [j E P R(i,j) = }} be the set of submissions assigned to agent and n2 = {i E N R(i, 1} be the set of agents assigned to review submission j. We want the assignment to be valid, i.c., satisfy the following constraints:
Each agent must be assigned at most ka submissions for review, i.e., |R < kas V.
Each submission must be assigned to kp agents, i.e., RR] kp. Vj P.
No agent should review one of her own submissions, i.e., R(i, = 0, N N [m.).
To ensure that a valid assignment always exists, we impose the constraint that m, ky < ka for each i E N, which implies that m Kn & n -a. Intuitively, this demands that each agent submitting papers be willing to provide as many reviews as the number of reviews assigned to the submissions of any single agent. For further discussion on this condition, see Section 5.
Note that given N' S N and P S P. for each i E N' with P' = UIEN' P/,, the validity requirements above can also be extended to a restricted assignment R N' x p' {0, 13. Hereinafter, we will assume validity unless specified otherwise or during the process of building an assignment.
Preferences. Each agent i E N has a preference ranking, denoted di,,, over the agents in N \ {i) for reviewing her ê-th submission pt.e.² These preferences can be based on mixture of many factors, such as how qualified the other agents are to review submission Pi,t, how likely they are to provide
*Our algorithms continue to work with weak orders; one can arbitrarily break ties to convert them into strict orders before feeding them algorithms.
3

---

positive review for it, etc. Let (t,) be the position of agent i' E EN {i} in the ranking. We say that agent prefers agent i' to agent i" as reviewer for P.. if o,,((t) o,,)("), Again, in the special case where the agents have single submission each, we drop l and just write dg Let (0,.............. On,1....,,n,....
While our algorithm takes ē as input, to reason about its guarantees, we need to define agent preferences over assignments by extending ō. In particular, an agent assigned a set of reviewers for each of her submissions, so we need to define her preferences over sets of sets of reviewers. First, we extend to preferences over sets of reviewers for a given submission, and then aggregate preferences across different submissions. Instead of assuming a specific parametric extension (e.g., additive preferences), we allow all possible extensions that satisfy two mild constraints; the group fairness guarantee of our algorithm holds with respect to any such extension.
Extension to set of reviewers for one submission: Let S > S' (resp., S F., S') denote that agent i strictly (resp.. weakly) prefers the set of agents 5 to the set of agents 5' for her f-th submission Pt,e- We require only that these preferences satisfy the following mild axiom.
Definition 1 (Order Separability). For every disjoint S1, S2, S3 S N with |S
[S2] > 0, if it holds that (i)) < ot,e(t") for each i' E S1 and i" E S2, then we must have S1 U. i S2 U S3.
An equivalent reformulation is that between any two sets of reviewers S and T' with |S| = (T), ignoring the common reviewers in S T, the agent strictly prefers every (even the worst) reviewer in S T to every (even the best) reviewer in T') \ SS, then the agent must strictly prefer $ to T.
Example Consider the common example of additive preferences, where each agent has a utility function u4 N \ {i} + Ez0 over individual reviewers for her f-th submission, inducing her prefer- ence ranking oi,-- In practice, these utilities are sometimes called similarity scores. Her preferences over sets of reviewers are defined via the additive utility function (s,e(S) & Eves et,)))"). It is easy to check that for any disjoint S2, S3 with |St| = SSal > O, ui,e(t') > ui,((t") for all i' E Si and i" E S2 would indeed imply "4 (S U Sg) u, (S2 US). Additive preferences are just one example from a broad class of extensions satisfying order separability.
Extension assignments. Let us now consider agent preferences over sets of sets of reviewers, or equivalently, over assignments. Let R >, R (resp., R b R) denote that agent strictly (resp.. weakly) prefers assignment R to assignment R. Note that these preferences collate the submission preferences i, across all submissions of the agent. We require only that the preference extension satisfies the following natural property.
Definition 2 (Consistency). For any assignment R, restricted assignment R over any N' S N and P' Pi (where Py S P, for each E N'), and agent 1" N', it holds that R P.r.e F
for each ,s-, E P. then we must have >; R. RS P..
In words, if agent weakly prefers R to R for the set of reviewers assigned to each of her submissions individually, then she must prefer R to R overall.
Example 2. Let us continue with the previous example of additive utility functions. The pref- erences of agent i can be extended additively to assignments using the utility function uy (R) EpPrp Mi.e( (Rf..) It is again easy to check that if u RPp.)) > Wi,ee eie ) for each Pr,e then (,(R)= > u, (R). Hence, additive preferences are again one example out of broad class of preference extensions that satisfy consistency.
Core. Our goal is to find group-fair assignment which treats every possible group of agents at least as well as they could be on their own, thus ensuring that no subset of agents has an incentive to deviate and set up their own separate conference. Formally:
Definition 3 (Core). An assigmment R is in the core if there is no N' C N. C P. for each i E N', and restricted assignment Ñ over N' and P UiEN P/' such that R >; R for each i E N'.
In words, if any subset of agents deviate with any subset of their submissions and implement any restricted reviewing assignment, at least one deviating agent would not be strictly better off, thus eliminating the incentive for such a deviation. We also remark that our algorithm takes only the preference rankings over individual reviewers o as input and produces an assignment R that is

---

review each ALGORITHM 1: CoBRA
Input: N.P.õ. P.ā,a.ã.a. k Rp
Output: R.
R. L, U =PRA-TTC(N, P,o, ka, kip);
if [U] 0 then
3
R Filling-Gapste N, P. lia kips R2 I, U);
ALGORITHM 2: PRA-TTC
Input: N, W. ka:
Output: R, L.U
1 R(i,j) < 0, Vi E N and Vj E P: Construct the preference graph Gn:
3 while 3 cycle in Gn do
Eliminate the cycle;
Update P.-s by removing any completely assigned paper:
Update Gn:
U E N #0
8 L + the last h [0] +
agents in N \ to have all their submissions completely assigned;
guaranteed to be in the core according to every preference extension of ā satisfying order separability and consistency.
3 CoBRA: An Algorithm for Computing Core-Based Reviewer Assignment
In this section, we prove our main result: when agent preferences are order separable and consistent, an assignment in the core always exists and can be found in polynomial time.
The main algorithm CoBRA "Cor-BBased Reviewer Assignment), presented as Algorithm 1, uses two other algorithms, PRA-TTC and Dilling-Gaps, presented as Algorithm 2 and Algorithm 3, respectively. We remark that PRA-TTC is adaptation of the popular Trading--Cyllee (TTC) mechanism, which is known to produce an assignment in the core for the house reallocation problem (and its variants) [22]. The adaptation mainly incorporates the constraints related to how many papers each reviewer can review and how many reviewers should review each paper. While for kp ka 1, PRA-TTC is identical with the classic TTC that is used for the house reallocation problem, the main difference of this problem with the review assignment problem is that in the latter each agent should give away her item (i.e. her submission) and obtain the item of another agent. Therefore, by simply executing TTC in the review assignment problem, one can get into a deadlock before producing valid assignment. For example, consider the case of three agents, each with one submission. Each submission must receive one review (kpp = 1) and each agent provides review (k., 1). The TTC mechanism may start by assigning agents 1 and other's submission, but this cannot be extended into a valid assignment because there is no one left to review the submission of agent 3. This is where Filling Gaps comes in; it makes careful edits the partial assignment produced by the PRA-TTC, and the key difficulty is to prove that this produces a valid assignment while still satisfying the core.
Techniques and key challenges:
3.1 Description of CoBRA
Before we describe CoBRA in detail, let us introduce some more notation. Let m mmx m,- For reasons that will become clear later, we want to ensure that m; m", for each i E N. To achieve that, we add m - m, dummy submissions to agent i, and the rankings over reviewers with respect these submissions are arbitrarily. An assignment called partial if there are submissions that are reviewed by less than kp agents. A submission that is reviewed by k, agents under a partial assignment is called completely assigned. Otherwise, it is called incompletely assigned. We denote with (,( the set of submissions of that are incompletely assigned under a partial assignment R. We omit from the notation when it is clear from context.
CoBRA calls PRA-TTC, and then if needed, it calls Filling-Gai Gapss Below, we describe the algorithms.
5

---

ALGORITHM 3: Filling-Gay
Input: N, N.P.o, ka: kip, R, L, U
Output: R
Phase
Construct the greedy graph Gn;
2 while cycle do
Eliminate the cycle;
Update P.-s by removing any completely assigned paper;
Update U and by moving any agent from U L if P.
Update GR:
Phase 2:
Construct the topological order ç of GRi
8 for t E [|4]]] do
while Po(e) + do
10
Pick arbitrary P.(t). E P P
11
Find completely assigned py,e, with R(p(t).pr.rt) 0, for some i' E GUUL) {p(t); Find 1" * p(t) such that R(i", .Pott),) 0 and R(i", pp R(i", (P((),e) < 1; R(i", .p,.)) t 0; R(p(t),Pxs.e) 1;
13
14
Remove Potti, from P Po()) if it completely assigned:
PRA-TTC. In order to define PRA-TTC, we first need to introduce the notion of a preference graph. Suppose have partial assignment R. Each agent i with P. * 0 picks of her incompletely assigned submissions arbitrarily. Without loss generality, we assume that she picks her (--th submission. We define the directed preference graph G (N, Eg) where each agent is node and for each i with P. * (i, E if and only is ranked highest in dze-- among the agents that don't 't review pi,e and review less than kn submissions. Moreover, for each i E N with P. 0. we add edge from i to i', where i' is arbitrary agent with P + 0. PRA-TTC starts with an empty assignment, constructs the preference graph and searches for directed cycle. such a cycle exists, the algorithm eliminates it as following: For each (i, that is included in the cycle, it assigns submission Pi,- to i' (if i's submissions are already completely assigned, it does nothing) and removes Pi,g- from P.. if it is now completely assigned. Then, the algorithm updates the preference graph and continues eliminate cycles in the same way. When there are left cycles in the preference graph, the algorithm terminates and returns two sets, U and L. The first set contains all the agents that some of their submissions are incompletely assigned and the L contains the last kp [0| + agents whose all submissions became completely assigned.
Filling-Gaps. CoBRA calls Filling- Gaps, if the U that returned from PRA-TTC is non empty. Before we describe the Aaling--apss, we also need to introduce the notion of a greedy graph. Suppose that we have a partial assignment R which indicates a set U that contains all the agents whose at least one submission is incompletely assigned. We define the directed greedy graph GR = (U, E
where (i, i' F if R(i', .p.e) 0 for some P., E P,. In other words, while in the preference graph, agent i points only to her favourite potential reviewer with respect to one of her incomplete submissions, in the greedy graph agent points to any agent in U {o} that could review at least one of her submissions that incompletely assigned. Filling-Gaps consists of two phases. In the first phase, starting from the partial assignment R that was created from PRA-TTC, it constructs the greedy graph, searches for cycles and eliminates a cycle by assigning Pi. to agent for each (") in the cycle that exists due to Pi,e (when an edge exists due to multiple submissions, the algorithm chooses one of them arbitrary). Then, it updates P. by removing any pi, that became completely assigned and also updates U by moving any i to L if P, became empty. It continues by updating the greedy graph and eliminating cycles in the same way. When no more cycles exist in the greedy graph, the algorithm proceeds to the second phase, where in |U] rounds ensures that the incomplete submissions of each agent become completely assigned.
In the appendix, there is an execution of CoBRA in a small instance.

---

3.2 Main Result
We are now ready to present our main result.
Theorem 1. When agent preferences are order separable and consistent, CoBRA returns an assign- ment in the core in On²) time complexity.
Proof. First, in the next lemma, we show that CoBRA returns a valid assignment. The proof is quite non trivial and several pages long, so we defer it to the supplementary material.
Lemma 1, CoBRA returns a valid assignment.
Now, we show that the final assignment R that CoBRA returns in the core. Note that while is possible that assignment of submission of an agent in U U L, that was established during the execution of PRA-TTC, to be removed in the execution Filling-Gaps, this never happens for submissions that belong to some agent in N (U U L). For the sake of contradiction, assume that N' N. with P. S P, for each i N', deviate to a restricted assignment over N' and U Note that R is valid only if | > kpp as otherwise there is no way each submission in U. to be completely assigned, since no agent review her submissions.
We distinguish into two cases and we show that in both cases the assignment is in the core.
Case I: 3i E N' : & LUU. Let i* E N' be the first agent in N' whose all submissions became completely assigned in the execution of PRA-TTC. Note that since there exists & U U L. we get that i" & U U L from the definitions of U and L. Now, consider any pe Let Q1 = R ppr .c. (R
P... n
RE . and Q2 = Re... pir.e n ....)) ). If Q1 = 0, then we have that R p.t. = R
p.., which
p. Otherwise, let i' = argmax o())(i), i.c. i' is ranked at the means that p.p... Fir RP
lowest position in or-,e among the agents that review .t under R but not under R. Moreover, let , ,()), i.e. i" is ranked the highest position in oe- among the agents that i" = ar:min 0y.
review p. e under R but not under R. We have R(i', p..,e)) 1, if and only if has outgoing edge to i' at some round of PRA-TTC. At the same round, we get that i" can review more submissions, since 1" E N' and if 1' has incompletely assigned submissions, then any i N' has incompletely assigned submissions, and hence R.S. < kp m & ka- This means that if 02 e((') > or e(t"),
then i" would point 1" instead of i'. We conclude that (i=))(()) De=,e(i") Then, from the definition of i' and i" and from the order separability property we have that RP... > RD pi- Thus, either
Q1 is empty or not, we have that for any Ps..e E P. it holds that RE pir.e Fe, Rp and from consistency, we get that R b R which is a contradiction.
Pitee
Case II Ai E N' : i & L U U. In this case we have that N' = UUL, as JUL kp + 1. This means that for each
UUU.U. and é E [m*), Re. (U (i). Let 1" T. be the first agent in L whose all submissions became completely assigned in the execution of PRA-TTC. Consider any er Note that it is probable that while pi-, e was assigned to some agent in PRA- TTC, was moved to another agent i' during the execution of Filling-Gaps Gaps. But, i' belongs to U and we can conclude that if pi is assigned to some E N \ U at the output of CoBRA, this assignment took place during the execution of PRA-TTC. Now, let Q1 Rg Rg.,, (R n Rg and Q2 Rg. (RR. Rg If 0. then we have that R$,-, Ri.., which means that RE.,, ,E R If Q1 + 0, then Qı N (UU L) and Q2 C UUU U L since Re.,, U U L. Let = arg os+,e()), i.e. i' ranked at the lowest position in o.-, among the agents that review Pe- , under R. but not under R. Moreover, let 1" argmin,gq; de+ (()), i.e. i" is ranked at the highest position in de among the agents that review .a*, under R but not under R. From above, we know that the assignment of Pi+,e l to i' was implemented during the execution of PRA-TTC, since d' E N \ (U U L). Hence, with very similar arguments as in the previous case, we will conclude that (i") < (i"). We have R(i'pss.. .e)
if and only if i* has an outgoing edge to some round of PRA-TTC. At this round, we know that i" can review more submissions, since i" E N' and if has incompletely assigned submissions, then any E N' has incompletely assigned submissions. This means that if or e(i') or e(i"), then i" would point z" instead of i'. Hence, we conclude that
7

---

or-e(ii) < <rr)((").. Then, from the definition of i' and i" and from the order separability property we have that Re i Ree p Thus, either if Qi is empty or not, we have that for any pi E P. it holds that R2,,, Fir, p..., and from consistency we get that R 3.- R which is a contradiction.
Lastly, we analyze the time complexity of CoBRA. First, we consider the time complexity of PRA- TTC. In each iteration, the algorithm assigns at least one extra reviewer to at least one incompletely- assigned submission. This can continue for at most - kip
Kig iterations, since each submission should be reviewed by kip reviewers. In each iteration, it takes (n) time to find and eliminate a cycle in the preference graph. Then, takes O(n²) time to update the preference graph, since for each cesttrarily-pickee
caommlleeel-assignaee submission of each agent, need to find the most qualified reviewer who can be additionally assigned to it. By all the above, we conclude that the runtime of PRA-TTC is O(n²), by ignoring ka which is small constant in practice. After PRA-TTC terminates, CoBRA calls the Filling-Gap algorithm. However, Lemma ensures that at the end of PRA-TTC, |LU < kp + 1, which is also small constant. And Filling-Gapp only makes local changes that affect these constantly many agents. As such, the running time of Filling-Gape is constant as well. Therefore, the time complexity of CoBRA is (
4 Experiments
In this section, empirically compare CoBRA to TPMS [1], which is widely used (for example, it was used by NeurlPS for many years), and PR4A [13], which was used in ICML 2020 [24]. As mentioned in the introduction, these algorithms assume the existence of similarity affinity score for each pair of reviewer and paper j. denoted by ji. The score (or utility) of a paper under an assignment Rt denoted by vyy is computed as uz _ Σieny (i,, TMPS finds an assignment R
that maximizes the utilitarian social welfare (USW), i.e., the total paper score Σjep udz whereas PR4A finds an assignment that maximizes the egalitarian social welfare (ESW), i.e., the minimum paper score m u.. We use ka- kp 3 in these experiments.
Datasets. We use three conference datasets: from the Conference on Computer Vision and Pattern Recognition (CVPR) in 2017 and 2018, which were both used by Kobren al. [16), and from the International Conference on Learning Representations (ICLR) in 2018, which was used by Xu et al. [25]. In the ICLR 2018 dataset, similarity scores and confticts of interest are also available. While a conflict between reviewer and a paper does not necessarily indicate authorship, it is the best indication we have available, so, following Xu et al. [25]. we use the conflict information to deduce authorship. Since in our model each submission has one author, and no author can submit more than [kg/ = 1 papers, we compute a maximum cardinality matching on the conflict matrix to find author- paper pairs, similarly to what Dhull et al. [26] did. In this way, we were able to match 883 out of the 911 papers. We disregard any reviewer who does not author any submissions, but note that the addition of more reviewers can only improve the results of our algorithm since these additional reviewers have no incentive to deviate. For the CVPR 2017 and CVPR 2018 datasets, similarity scores was available, but not the conflict information. In both these datasets, there are fewer reviewers than papers. Thus, we constructed artificial authorship relations by sequentially processing papers and matching each paper to the reviewer with the highest score for it, if this reviewer is still unmatched, In this way, we were able to match 1373 out of 2623 papers from CVPR 2017 and 2840 out of 5062 papers from CVPR 2018. In the ICLR 2018 and CVPR 2017 datasets, the similarity scores take values in [o, 1]. so we accordingly normalized the CVPR 2018 scores as well.
Measures. We are most interested in measuring the extent to which the existing algorithms provide incentives for communities of researchers to deviate. To quantify this, we need to specify the utilities of the authors. We assume that they are additive, i.e., the utility of each author in an assignment is the total similarity score of the hp = 3 reviewers assigned to their submission.
'Technically, subject to this, it maximizes the second minimum paper score, and then the third minimum paper score, etc. This refinement is also known as leximin in the literature.
"Note that k, = reviews per submission is quite common, although the reviewer load k. is typically higher in many conferences, often closer to 6. However, the differences between different algorithms diminish with higher values of k..
8

---

a-Core
Dataset
Alg
USW
ESW
CV-Pr
#unb-a
a²
CoBRA
1.225 + 0.021
0.000
0.000
0%
1.000 0.000
0%
CVPR '17
TPMS
1.497 = 0.019
0.000 0.000
89%
3.134 0.306
100%
PR4A
1.416 + 0.019
0.120 0.032
51%
1.700 0.078
100%
CoBRA
0.224 + 0.004
0.004
0.001
0%
1.000 0.000
0%
CVPR '18
TPMS
0.286 + 0.005
0.043 = 0.004
0%
1.271
0.038
100%
PR4A
0.282 = 0.005
0.099 = 0.001
0%
1.139
0.011
100%
CoBRA
0.166 + 0.001
0.028
0.001
0%
1.000
0.000
0%
ICLR '18
TPMS
0.184 + 0.001
0.048 = 0.002
0%
1.048
0.008
90%
PR4A
0.179 = 0.001
0.082 = 0.001
0%
1.087 0.009
100%
Table Results on CVPR 2017 and 2018, and ICLR 2018.
Core violation factor: Following the literature [27]. measure the multiplicative violation of the core (if any) that is incurred by TPMS and PR4A (CoBRA provably does not incur any). This is done by computing the maximum value of by deviating and implementing some valid reviewing assignment of their papers among themselves, they can each improve their utility by factor of at least a. This can easily be formulated as a binary integer linear program (BILP). Because this optimization is computationally expensive (the most consuminn component our experiments), we subsample 100 papers from each dataset in each run, and report results averaged over 100 runs. Note that whenever there exists subset of authors with zero utility each the current assignment who can deviate and receive positive utility each, the core deviation a becomes infinite. We separately measure the percentage of runs in which this happens (in the column #unb-a), and report the average a among the remaining runs in the a* column.
for which there exists a subset of authors such that
Core violation probability: We also report the percentage of runs in which core violation exists (i.e., there exists at least one subset of authors who can all strictly improve by deviating from the current assignment). We refer to this as the core violation probability (CV-Pr).
Social welfare: Finally, we also measure the utilitarian and egalitarian social welfare (USW and ESW) defined above, which are the objectives maximized by TPMS and PR4A, respectively.
Results. The results are presented in Table As expected, TPMS and PR4A achieve the highest USW and ESW, respectively. all datasets because they are designed to optimize these objectives. In CVPR 2017, CoBRA and TPMS always end up with zero ESW because this dataset includes many zero similarity scores, but PR4A is able to achieve positive ESW. In all datasets, CoBRA achieves a relatively good upproximation with respect to USW, but this is not always the case with respect to ESW. For example, in CVPR 2018, CoBRA achieves 0.004 ESW achieves 0.099 ESW on average. This may be due to the fact that this dataset also contains many zero similarity scores, and the myopic process of CoBRA locks itself into bad assignment, which the global optimization performed by PR4A avoids.
average whereas PR4A
While CoBRA suffers some loss in welfare, TPMS and PR4A also generate significant adverse incentives. They incentivize at least one community to deviate in almost every run each dataset (CV-Pr). While the magnitude of this violation is relatively small when it is finite (except for TPMS CVPR 2017), TPMS and PR4A also suffer from unbounded core violations in more than half of the runs for CVPR 2017; this may again be due to the fact that many zero similarity scores lead to deviations by groups where each agent has zero utility under the assignments produced by these algorithms.
Of all these results, the high probability of core violation under TPMS and PR4A is perhaps the most shocking result; when communities regularly face adverse incentives, occasional deviations may happen, which can endanger the stability of the conference. That said, CoBRA resolves this issue at significant loss in faimess (measured by ESW). This points to the need for finding a middle ground where adverse incentives can be minimized without significant loss in fairness or welfare.
'In Table 2 in the appendix, we report USW and ESW without subsampling and we note that the qualitative relationships between the different algorithms according each metric remain the same.

---

Discussion
In this work, we propose a way for tackling the poor reviewing problem in large conferences by introducing the concept of "core" notion of group fairness in the peer review process. This faimness principle ensures that each subcommunity is treated least as well as would be if was not part of the larger conference community.
We show that under certain -albeit sometimes in the core always exists and can be efficiently found. In the experimental part, we provide evidence that peer review assignment procedures that are currently used in practice, quite often motivate subcommunities to deviate and build their own conferences.
peer review assignment
Our theoretical results serve merely as the first step toward using it find reviewer assignments that treat communities fairly and prevent them from deviating. As such, our algorithm has significant limitations that must be countered before is ready for deployment in practice. A key limitation is that it only works for single-author submissions, which may be somewhat more realistic for peer review of grant proposals, but unrealistic for computer science conferences. We also assume that each author serves as a potential reviewer; while many conferences require this nowadays, exceptions must be allowed in special circumstances. We also limit the number of submissions by any author to be at most [ka/kp], which is a rather small value in practice, and some authors ought to submit more papers than this. We need make this assumption to theoretically guarantee that valid assignment exists. An interesting direction is to design an algorithm that can produce a valid assignment in the (approximate) core whenever exists. Finally, deploying group fairness in real-worde peer review processes may require designing algorithms that satisfy approximately at minimal loss in welfare, as indicated by our experimental results.
References
[I] Laurent Charlin and Richard Zemel. The toronto paper matching system: an automated paper- reviewer assignment system. In Proceedings of the ICML Workshop on Peer Reviewing and Publishing Models, 2013.
[2] G. David L. Travis and Harry M. Collins. New light on old boys: Cognitive and institutional particularism in the peer review system. Science, Technology, & Human Values, 16(3):322 341, 1991.
[3] Raymond S. Nickerson. What authors want from journal reviewers and editors. American Psychological, pages 661-666 2005.
[4] Nihar B. Shah. Challenges, experiments, and computational solutions in peer review. Commun. ACM, 65(6))76-87, 2022.
[5] David Mimno and Andrew McCallum. Expertise modeling for matching papers with reviewers. In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 500-509, 2007.
[6] Xiang Liu, Torsten Suel, and Nasir Memon. A robust model for paper reviewer assignment. In Proceedings of the Sth ACM Conference on Recommender systems, pages 25-32, 2014.
[7] Marko A. Rodriguez. and Johan Bollen. An algorithm to determine peer-reviewers. In Proceed- ings of the 17th ACM conference on Information and knowledge management, pages 319-328, 2008.
[8] Hong Diep Tran, Guillaume Cabanac, and Gilles Hubert. Expert suggestion for conference program committees. In 2017 11th International Conference on Research Challenges in Information Science (RCIS), pages 221-232. IEEE, 2017.
[9] Andi Peng, Jessica Zosa Forde, Yonadav Shavit, and Jonathan Frankle. Strengthening sub- communities: Towards sustainable growth in ai research. arXiv preprint arXivv 2022.
[10] Donald Bruce Gillies. Some theorems n-person games. Princeton University, 1953.
10

---

[11] Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances neural information processing systems, 29, 2016.
[12] Ivan Stelmakh, John Wieting, Graham Neubig, and Nihar B Shah. A gold standard dataset for the reviewer assignment problem. arXiv preprint arXiv:2303. 16750, 2023.
[13] Ivan Stelmakh, Nihar B. Shah, and Aarti Singh. Peerreview4all: Fair and accurate reviewer assignment in peer review. In Algorithmic Learning Theory, pages 228-856. PMLR, 2019.
[14] Regina O'Dell, Mirjam Wattenhofer, and Roger Wattenhofer. The paper assignment problem. Technical Report/ETH Zurich, Department of Computer Science, 491, 2005.
[15] David Hartvigsen, Jerry C Wei, and Richard Czuchlewski. The conference paper-reviewer assignment problem. Decision Sciences, 30(3):865-88) 876, 1999.
[16] Ari Kobren, Barna Saha, and Andrew McCallum. Paper matching with local fairesss constraints. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1247-1257, 2019.
[17] Justin Payan and Yair Zick. I will have order! optimizing orders for fair reviewer assignment. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 440-446, 2022.
[18] Duncan Karl Foley. Resource allocation and the public sector. Yale University, 1966.
[19] Naveen Garg, Telikepalli Kavitha, Amit Kumar, Kurt Mehlhorn, and Julián Mestre. Assigning papers to referees. Algorithmica, 58(1):1 19-136, 2010.
[20] Jing Wu Lian, Nicholas Mattei, Renee Noble, and Toby Walsh. The conference paper assignment problem:
Using order weighted averages to assign indivisible goods. Sheila A. Mcllraith and Kilian Q Weinberger, editors, Proceedings the Thirty-Secone Second AAAI Conference Artificial Intelligence (AAAI-18), pages 1138-11110 1145. AAAI Press, 2018.
[21] Nihar B. Shah. Challenges, experiments, and computational solutions in peer review. Communi- cations of the ACM, 65(6):76-87, 2022.
[22] Lloyd Shapley and Herbert Scarf. On cores and indivisibility. Journal of Mathernatical Economics, 1(1):23-37, 1974.
[23] Takamasa Suzuki, Akihisa Tamura, and Makoto Yokoo. Efficient allocation mechanism with endowments and distributional constraints. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), pages 50-58, 2018.
[24] Ivan Stelmakh. Towards fair, equitable, and efficient peer review. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 15737-15777, 2021.
[25] Yichong Xu, Xiaofei Zhao, Han Shi, and Nihar B. Shah, On strategyproof conference peer review. In Proceedings of the 28th International Joint Conference Artificial Intelligence (IJCAI), pages 616-622, 2019.
[26] Komal Dhull, Steven Jecmen, Pravesh Kothari, and Nihar B. Shah. Strategyproofing peer assessment via partitioning: The price in terms of evaluators' expertise. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, pages 53-63, 2022.
[27] Brandon Fain, Kamesh Munagala, and Nisarg Shah. Fair allocation of indivisible public goods. In Proceedings of the 2018 ACM Conference on Economics and Computation, pages 575-592, 2018.
11


Reviews:
[{"review_id": "Yc1MtN67dk", "reviewer": "Reviewer_VPuM", "summary": "The authors frame their investigation into group fairness in the peer review setting in the context of large conferences (i.e. NeurIPS, AAAI) by considering a simplified peer review model that enforces the existence of a valid reviewer assignment.\n\nWithin this framework, the authors apply the fairness notion of \"the core\" to this setting and present an efficient graph-based algorithm that always returns a valid assignment in the core under minor conditions based on author preferences. \n\nThe authors empirically validate their method using real data from CVPR and ICLR, and evaluate the cost of utilizing this algorithm (in terms of lost utilitarian and egalitarian welfare) in order to satisfy the fairness notion of \"the core\" and prevent the incentive for any community to establish their own separate smaller conference (which can lead to research topic insularity and harm interdisciplinary research areas).", "questions": "1. You mention that theoretically, the CoBRA algorithm is efficient (polynomial time). In terms of actual compute time, how does this scale as a function of the dataset size? Can you provide estimates for the wallclock compute time for the empirical analysis?\n\n2. Why is the introduction of the notion of the core meaningful for the peer reviewer assignment problem space? The Related Work included highlights some relevant work in this area, but does not (from my perspective) very clearly or succinctly motivate the open problem you're addressing in a compelling way. This makes it challenging to determine the potential impact of this work. \n\n3. [A \"nice to have\" suggestion around presentation] While _technically_ the mathematical definitions of measures computed in empirical evaluation are included in Section 4, they are buried within the main text body; these may be easily missed by the reader or make it harder to understand the paper (it appears that main paper real estate was scarce). I suggest that the authors make room in the main paper to describe the measures used mathematically and verbally (at least prior to publishing) and/or expand upon these along with additional rationale/significance in the Appendix.", "limitations": "Practical limitations are well outlined in the final section of the paper, along with directions for follow-up work to build upon this theory-forward paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- While there is a good bit of notation in the model presented, it is kept fairly simple and is clearly described/understandable (to me). This is significant because the paper's primary contributions are methodological and theoretical.\n\n- The authors do well in presenting mathematical definitions will more intuitive descriptions, lowering the cognitive barrier on the reader. \n\n- The proof provided for main theoretical finding (Theorem 1, along with subsequent lemmas and prop) is very rigorous, and thorough. \n\n- Toy example illustrating the execution of the algorithm introduced by the authors (CoBRA) is helpful for building intuition of the method (adding a graph-based visualization of the two companion graphs would be a bonus).\n\n- The empirical results and interpretation are well-described in words and may benefit from modified format to enabled more direct comparison between methods and highlight potential performance trade-offs (please refer to suggestion in \"Weaknesses\" section on elaboration).\n\n- The authors humbly acknowledge the limitations of their work before it is ready to be used in practice and highlight some potential follow-up directions for investigation.\n\n- Overall comment: much of the content is clearly described, but the presentation structure adds cognitive overhead on the reader and makes it more challenging to understand and utilize the provided information. There are clear strengths in this category, but also real room for improvement. [I struggled to determine the rating in this category, perceived to be between a 2-3 overall. Please refer to \"Weaknesses\" and \"Questions\" for more details.]", "weaknesses": "Primarily, feedback is around presentation and experimental results/details. Some of these are more significant than others. Overall, the paper is fairly well-written but appears very cramped; the structure and presentation could be improved to support understandability and readability.\n\n- Additional details on empirical setup and experiments is necessary to support reproducibility. (Note that there is a line in the Appendix outlining the computation resources and some core code files are included in the supplementary.) \n\n- Aligned with the comment under \"Strengths\", there would be benefit in modifying/better aligning the quanititative results with the discussion/explanation (in Section 4 -> Results). The content is good, but presentation is a little hard for me to follow and has room for improvement (perhaps by highlighting best performance for each measure/column in Table 1?). When page limit is not as tight, it'd be helpful to reorganize Section 4 into subsections (and similarly in Section 4). \n\n- It's worth noting that _what_ the authors present is fairly clear, but a clearer motivation of the open problem they are working to address is lacking. (Follow-up question in \"Questions\" section.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for providing useful recommendations for improving the readability of our paper. We will incorporate them in our revision.\n\n### **Q1: Time complexity**\n\nThe worst-case time complexity of CoBRA is $O(n^3)$ and we have provided the average-case runtime in the PDF attached to the common response. Please see our common answer to all the reviewers for more details. \n\n### **Q2: Motivation behind the core in peer review**\n\nPlease see our common answer to all reviewers. "}, {"review_id": "vP0Ismd5na", "reviewer": "Reviewer_YGAX", "summary": "This paper proposed an approach that lets the assignment of the peer review model satisfy the core, a fairness requirement over groups of authors, so as to prevent small research communities from having the incentive to deviate and set up their own separate conferences. Through theoretical analysis, the authors found that the proposed method, CoBRA, would return a valid assignment in the core if agent preferences are order separable and consistent. Experimental results show that the proposed CoBRA can produce much fair assignments compared to the baseline approaches.", "questions": "1. How optimal is CoBRA's assignment?\n2. How does each author's paper submission amounts affect the fair peer review assignment made by CoBRA? Some areas will receive a huge number of submissions but the other areas may not. The number of submissions by different authors also varied a lot. How does CoBRA address such data imbalance issues and guarantee group fairness?\n3. How to determine groups in the experiment?", "limitations": "The limitations have been adequately discussed.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper addressed an important problem, ensuring group fairness in the peer review process.\n2. The proposed CoBRA is theoretically and technically sound.\n3. As a theoretical paper, it is not difficult to follow. Several examples and cases are given for the audience to better understand the theoretical analysis as well as the proposed models.", "weaknesses": "1. The necessity of achieving fairness in peer review has not yet been well motivated. It is unclear why achieving group fairness will result in the improvement of satisfaction for various communities.\n2. The proof of Theorem 1 is not complete. Lemma 1 only validates CoBRA can return the valid assignment in the core but the theoretical analysis on the time complexity of CoBRA to generate the assignment is not complete.\n3. The optimality of the valid assignment is not proven.\n4. The flexibility of putting the CoBRA in practical use has not been fully discussed. I suggest the authors discuss how CoBRA could deal with the imbalance submission when putting in practical use.\n5. It is not easy for non-expert audience to understand this paper, especially why satisfying core is good enough for the review assignments.\n\nSome minor suggestions:\n1. The proposed methods can be extended to scheduling tasks in many other application scenarios such as task scheduling in collaborative edge computing, federated learning, etc. Merely producing a fair per-review assignment may limit the contributions of the CoBRA.\n2. The technical challenges can move to the introduction sections.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your review. Please see our common answer to all reviewers for motivations behind the core as well as the exact running time of CoBRA.  \n\n### **Q1 & Q3 (and related comments)**\n\n> How optimal is CoBRA's assignment?\n\n> The optimality of the valid assignment is not proven.\n\n> How to determine groups in the experiment?\n\nWe believe that these comments may have stemmed from a fundamental misunderstanding regarding our objective, the core. \n\nAt its heart, the core is a *qualitative* notion of fairness; a reviewing assignment is either *in the core* or *not in the core*. There is no notion of “optimality”. One can say that approximation of the core turns this into a quantitative objective, but we prove that our algorithm finds an assignment *in the core* (i.e., achieves the best possible 1-approximation of the core). Thus, there is no optimality to be proven. \n\nAlso, the main benefit of the core is that it simultaneously provides a fairness guarantee for *every possible group* (that is, all $2^n-1$ non-empty subset of agents). As such, unlike most definitions of fairness considered in the machine learning literature, where one must specify groups in advance and fairness can only be achieved with respect to those groups, finding a reviewing assignment in the core achieves fairness with respect to all possible groups simultaneously. This also eliminates the need for determining groups in the experiments. \n\n### **Q2**\n\n> Some areas will receive a huge number of submissions but the other areas may not. The number of submissions by different authors also varied a lot. How does CoBRA address such data imbalance issues and guarantee group fairness?\n\nThis is again addressed by the definition of the core, which the assignment returned by CoBRA satisfies. The fairness guarantee offered by the core to every group depends on what the group can achieve on its own. As such, a group that consists of very few people but generates a large number of submissions may not be offered a strong guarantee because such a group, on its own, would not be able to generate high-quality reviews for so many submissions of its own. On the other hand, a group which produces only as many submissions as it can review well on its own and has the internal expertise to produce high-quality reviews for these submissions will find that CoBRA’s assignment treats it quite well, no worse than the satisfactory reviewing outcome it can produce on its own. \n\nWe find that this is a natural way in which the core handles imbalances between individuals or research areas without relying on any free parameters (deciding which can become quite controversial in practice). "}, {"review_id": "0Zf20Nrsb4", "reviewer": "Reviewer_E8Ux", "summary": "The authors propose to use core concept in the context of peer review system. Potentially decreasing an overall welfare, the new paradigm offers a fairer treatment of small sub-communities, erasing an incentive to create an independent venue. The paper provides an algorithm for assigning the reviewers on a restrictive case of single author submission. Experimental support their claims.", "questions": "It would be nice if the authors could write a more general model and definition (not only single author) for future references, while then saying that for now they only treat a particular case as a proof of concept.", "limitations": "Section 5 on limitations is pretty clearly written and the list seems exhaustive.\n\nHaving purely technical background, I do not feel qualified to comment on the main premise of the work: make smaller communities stay within gigantic conferences. At least in TCS and theoretical ML there are some great smaller conferences (SoCG, SOSA, COLT, ALT, FaccT to name a few). I am not sure that these communities would benefit from being incentivised to stay within say NeurIPS. In particular, it seems to me that fairness and accountability community has greatly benefited from the creation of FaccT (the most recent example I know). So, it would be nice to hear an opinion from people that submit regularly to smaller conferences.\n\nThat being said, I am positive about this submission as it tries to tackle a rather complicated, ill posed and timely problem. ", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The authors are trying to address a crucial problem in the modern era ML community---how to assign papers fairly and keep more communities happy. Overall I enjoyed reading the paper, found the notion of core interesting and the proposed methodology sound.", "weaknesses": "The authors are pretty up-front with listing the weaknesses of the work. The major one, of course, being the single-author case. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your useful feedback. As per your suggestion, we will define the core more generally in our revision for the benefit of future work, especially given that it is indeed a concept that applies quite broadly. \n\nPlease also see our common response to all the reviewers regarding your comment on the advantages of having small conferences. "}, {"review_id": "dY27PBDKiI", "reviewer": "Reviewer_HKZ7", "summary": "The authors consider a problem of finding reviewer assignments for conference peer review. Specifically, they aim to find a valid reviewer assignment subject to the constraint that no group of authors can achieve a (strictly) preferred reviewer assignment among themselves and a subset of their authored papers. This can be seen as both a fairness constraint and an incentive-compatibility constraint. The authors provide an algorithm to find such a reviewer assignment and theoretically prove that the returned assignment satisfies the desired properties. They then empirically compare the proposed algorithm to several baselines on subsampled real conference data, and show that it achieves reasonable total welfare while other baselines often violate the core constraint.", "questions": "- Could the authors clarify the time complexity of the algorithm?\n- I assume that the USW results in Table 1 are average assigned similarity, not total similarity as defined.", "limitations": "The authors clearly stated the limitations of their algorithm, noting that their contribution was primarily conceptual and not practical.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The problem of finding a paper assignment in the core within the peer review context is interesting and new (to my knowledge). I think the core constraint is well-motivated as a group fairness constraint and is substantively different from the other notions of fairness considered in prior work. \n- The writing is generally very clear and the paper is easy to read.\n- The author preference model used by the paper is significantly more general than the standard additive utility assumed in other works.  \n- The empirical results that standard TPMS assignments have significant numbers of core violations on real conference datasets are of practical interest.\n", "weaknesses": "- The setting assumed by the paper is highly simplified compared to the standard peer review setting: all papers are authored by a single author, and no conflicts-of-interest other than authorship are considered. This hinders the practical relevance of the algorithm. \n- While the proposed algorithm is claimed to be polynomial-time (Theorem 1), this claim does not seem to be proved and no analysis of the time complexity is provided.\n- The empirical results are constructed by subsampling small sets of reviewers and papers. While this is done for computational reasons when evaluating core violations, I’m not sure why the social welfare results were also approximated.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your useful feedback. \n\n### **Regarding subsampling**\n\nIndeed, we subsampled sets of reviewers and papers throughout our experiments for consistency (as you noted, this was required for evaluating core violations). If you are interested, the results for USW and ESW without any subsampling are in the table below. It is comforting to note that the qualitative relationships between the different algorithms according to each metric remains the same. It is worth noting that USW goes up across the board while ESW goes down for CoBRA and TPMS but up for PR4A in this case as opposed to the subsampling case from the submission. Note that theoretically, each of USW and ESW can go either up or down when we consider a superset of the data. \n\nWe would be happy to add this table to the appendix in our revision, if you wish us to. \n\n\n| Dataset | Algo    | USW   | ESW   |\n|---------|---------|-------|-------|\n|         | CoBRA   | 1.644 | 0.000 |\n| CVPR'17 | TPMS    | 1.970 | 0.000 |\n|         | PR4A    | 1.919 | 0.384 |\n|---------|---------|-------|-------|\n|         | CoBRA   | 1.208 | 0.000 |\n| CVPR'18 | TPMS    | 1.586 | 0.004 |\n|         | PR4A    | 1.560 | 0.731 |\n|---------|---------|-------|-------|\n|         | CoBRA   | 0.251 | 0.015 |\n| ICLR'18 | TPMS    | 0.284 | 0.038 |\n|         | PR4A    | 0.278 | 0.086 |\n\n### **Q1: Time complexity**\n\nPlease see our common response to all the reviewers for the time complexity.\n\n### **Q2: USW**\n\nYes, we apologize; these are averaged assigned similarities. We used average rather than sum to report USW and ESW on the same scale for better comparison. We will revise the text accordingly. \n"}, {"review_id": "8xH8XD2nGo", "reviewer": "Reviewer_wnxf", "summary": "The paper describes a new algorithm for assigning papers to peer reviewers with the goal of ensuring that review assignments are in the core. Thie motivation behind this is that large conferences benefit from joining many different communities of research and thereby enabling more interdisciplinary cooperation. If review assignments are in the core then subgroups will have no incentive to break off and form smaller, more siloed conferences.\n\nAn algorithm is described that, under certain assumptions, creates assignments between papers and authors that are within the core and claims to run in polynomial time. This algorithm is compared with two commonly used algorithms on a number of metrics. While CoBRA, the new algorithm, does not provide as much utility as the TPMS and PR4A algorithms it is shown that the other algorithms almost always produce outcomes that are not in the core. The paper concludes by suggesting that some middle ground between these approaches may be a useful path forward.", "questions": "Can you clarify what the X/Y matching is that you are referring to in the Datasets section? Does this indicate that you are giving as input to the algorithm only X out of Y papers from a conference?\n\n\nHow large do the deviating groups for TPMS and PR4A tend to be? If they are extremely tiny or nearly the entire group I would expect that to be a less meaningful deviation than if they are some distinct subfield of research within the conference.", "limitations": "The authors have done a very good job of discussing their limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "While the problem described by the paper has received a fair bit of research the application of the core appears to be a novel approach to the review assignment problem. The problem is not of critical importance but is one that has room for improvement and can relatively easily benefit from algorithmic enhancements.\n\nOverall the paper provides clear high-level descriptions of each step and the motivation behind why the authors believe the core is a useful idea for review assignment. The authors are also clear about the limitations of their work and where it does and does not improve upon the status quo.\n\nThe experiment section was a useful addition. Despite the results not dominating other algorithms I am glad there is some evaluation of CoBRA.", "weaknesses": "While the high level components of the paper are explained quite clearly I found the specific details in need of clarification. A number of grammatical issues exist (often across longer sentences; some issues on lines 8, 209, 226-229, 232, 256) Attempting to reformulate some technical ideas and write with simple, shorter sentences might ease reader comprehension.\n\nYour motivation is clearly described but I am not wholly convinced that smaller, more focused venues are generally worse. While they may be less naturally suited for interdisciplinary work they also (intuitively, to me) increase the chance that reviewers and researchers come across work more connected to the areas they are working in.\n\nWhile you claim there are only mild assumptions of order separability and consistency there appear to be a number of modeling assumptions that may not be realistic. You say that each author must serve as a reviewer, which (as you do mention) is not possible to guarantee. You also say that CoBRA \"takes as input only the preference ranking of each author over individual potential reviewers\" (line 58/59). Perhaps this is common phrasing in for reviewer assignment work but *as phrased* this seems like an extremely strong requirement. Being more clear that you probably mean something like a similarity score would be an improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe thank you for your useful feedback. We will incorporate all your suggestions in our revision. Please see our common answer to all reviewers with regards to your comment about the benefits of focused venues (which we fully agree with). \n\nAs for the inputs, we meant to say that we do not *need* numerical similarity scores, only ordinal comparisons (which can always be induced if you have access to similarity scores). But we’ll rephrase this for clarity. \n\n### **Q1: X/Y matching**\n\nYes, this is what we mean. We (like Xu et al. [25] and Dhull et al. [26]) are forced to do this because the datasets are anonymized whereas we need to connect the author set to the reviewer set. Thus, we use the conflict information to deduce authorship as best as we can, which is not a perfect process. \n\n### **Q2: Size of deviating groups**\n\nWe agree that the size of the deviating groups is an interesting measure, which we did not consider. In the table below, you can see the *maximum size* of a successfully deviating coalition, averaged across 100 runs, together with the standard error. Recall that each run is a subsampled dataset of size 100, so these can be interpreted as percentages. \n\nIt seems that under both TPMS and PR4A across all three datasets, the largest deviating communities are 6-15% of the conference size, which we believe can indeed reflect the sizes of some of the largest subcommunities at CVPR and ICLR. (Note that these are the largest deviations and there are smaller deviations too.) \n\nWe will be happy to include this in our revision. \n\n\n| Dataset  | TPMS      | PR4A      |\n|----------|-----------|-----------|\n| CVPR '17 | 6.64±0.77 | 7.5±0.77  |\n| CVPR '18 | 10.54±1.29| 11.49±1.49|\n| ICLR '18 | 11.25±1.76| 15.01±1.76|"}]

Author rebuttal:

We thank all the reviewers for their effort and for providing helpful reviews. We will be happy to incorporate all the suggestions of the reviewers as explained in more detail in the individual responses. Let us address two comments raised by multiple reviewers in this common response.

### **Motivation behind the core in peer review**

There are three key motivations for studying the core in the peer review setting. 

1) **Fairness:** The core acts as a notion of group fairness, which provides a guarantee that every possible group (even a community that doesn’t yet have a well-established identity) will be treated well relative to what the group can achieve on its own, which is a function of the reviewing burden imposed by the group versus the reviewing capacity contributed by the group.

2) **Stability:** The core also acts as a notion of stability because it ensures that no group would have an incentive to break off due to a feeling of being mistreated by the large conference (such as NeurIPS). Note that communities may still prefer to set up their specialized conferences for very good reasons including those outlined by the reviewers, but we believe that receiving low-quality reviews should not be one of them. This is precisely what the core aims to ensure. 

   We also remark that while specialized conferences have their advantages, so do large conferences. For example, they can find the diverse reviewing expertise needed for emerging multidisciplinary areas and provide a venue for interdisciplinary dialogues to take place. Thus, it is useful to retain at least some large conferences alongside specialized conferences, but this would be difficult if various communities keep breaking off due to not receiving high-quality reviews. 

   In this sense, we don’t see the core as creating roadblocks to the creation of specialized conferences, but rather as a way of mitigating harm imposed on communities in large conferences, thereby maintaining their existence alongside more specialized venues.  

3) **Robustness:** A key benefit of the core is that it is a robust definition of fairness that does not require specifying groups in advance, unlike most definitions studied in the machine learning literature. Instead, it simultaneously provides a fairness guarantee for *every possible group* (that is, all $2^n-1$ non-empty subset of agents); this includes groups defined based on sensitive attributes, intersectional groups, or even groups that do not yet have a well-established identity. Further, the guarantee scales in a principled manner, without having to set any controversial parameter values, as a function of what each group can achieve on its own, as mentioned above. 

We will clarify these more explicitly in our revision and revise language in our paper which, unfortunately, seems to incorrectly suggest that “preventing” small communities from deviating is our intention (it is not!). 

### **Running time of CoBRA**

We apologize for not including a running time analysis of CoBRA. Since we show that the number of assigned reviews grows monotonically (with each increase evidently requiring polynomially many steps), we believed it was immediate that the algorithm runs in polynomial time, which is all that Theorem 1 claims. Like many theoretical works, we were not overly concerned with the exact time complexity. However, we see that it is useful to not only lay out the polynomial time argument, but to also identify the exact complexity so future works can perhaps improve on it. We do so here and will include this in our revision. 

First, let us consider the time complexity of PRA-TTC. In each iteration, the algorithm assigns at least one extra reviewer to at least one incompletely-assigned submission. This can continue for at most $m \cdot k_p \leq n*k_a$ iterations, where the inequality follows from our condition for ensuring the existence of a feasible reviewing assignment. In each iteration, it takes $O(n)$ time to find and eliminate a cycle in the preference graph and $O(n^2)$ time to update the preference graph (for an arbitrarily-picked incompletely-assigned submission of each agent, we need to find the most qualified reviewer who can be additionally assigned to it). Thus, in total, the runtime of PRA-TTC is $O(n^3)$ because $k_a$ is a small constant in practice. 

After PRA-TTC terminates, CoBRA calls the Filling-Gaps algorithm. However, Lemma 3 ensures that at the end of PRA-TTC, $|L \cup U| \le k_p+1$, which is a small constant. And Filling-Gaps only makes local changes that affect these constantly many agents. As such, the running time of Filling-Gaps is constant as well. 

Overall, the time complexity of CoBRA is $O(n^3)$. 

For the average-case runtime, we have attached a PDF to the common response, which shows the running time of CoBRA as a function of the number of submissions in the conference. Results are obtained by subsampling submissions from the three datasets using the same process as in our experiments section, and average runtime over 25 runs is shown together with the standard error. Across all datasets, CoBRA runs in less than half a minute even with 800 submissions.

We will be happy to include this figure in the appendix in our revision. 

Meta-review: The paper defines a group fairness notion for the peer review assignment problem using a concept of core from economics where the goal is to find an assignment that treats every possible group as well as they could be on their own. The paper presents an efficient (polynomial time) graph-based algorithm that always returns a valid assignment under certain conditions on author preferences. The reviewers appreciate the timeliness and significance of the problem, as well as the soundness of the algorithm and the rigor with which the theoretical and empirical results are presented. The problem statement, the novel notion of fairness and the presented solution makes this a strong paper for NeurIPS. However, the reviewers noted a few things that I would encourage the authors to add to the final version. First, the submission should include a time complexity analysis of the solution (even if it can be easily inferred). Second, clarify the misunderstanding that the goal of the fairness notion is not for the paper authors to not breakaway to form a special conference, but for them to get good reviews from relevant reviewers. Finally, the authors should clearly state other practical limitations of the approach that arise due to the assumptions. This will ensure that future work on open questions and allow for adoption of better peer-reviewing assignment algorithms or practices.
Decision: Accept (spotlight)