SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haris Aziz
UNSW Sydney
haris.aziz@unsw.edu.au
Evi Micha
University of Toronto, Icha@cs.toronto.ca
Large conferences such as NeurIPS and AAAI serve as crossroads of various AI fields, since they attract submissions from a vast number of communities. However, in some cases, this has resulted in a poor reviewing experience for some communities, whose submissions get assigned to less qualified reviewers outside of their communities. An often-advocated solution is to break up any such large conference into smaller conferences, but this can lead to isolation of communities and harm interdisciplinary research. We tackle this challenge by introducing a notion of group fairness, called the core, which requires that every possible community (subset of researchers) to be treated in a way that prevents them from unilaterally benefiting by withdrawing from a large conference.
We study a simple peer review model, prove that it always admits a reviewing assignment in the core, and design an efficient algorithm to find one such assignment. We use real data from CVPR and ICLR conferences to compare our algorithm to existing reviewing assignment algorithms on a number of metrics.
Due to their large scale, conferences like NeurIPS and AAAI use an automated procedure to assign submitted papers to reviewers. Popular such systems include the Toronto Paper Matching System [1], Microsoft CMT<sup>1</sup>, and OpenReview<sup>2</sup>. The authors submitting their works are often very interested in receiving meaningful and helpful feedback from their peers [2-4]. Thus, their overall experience with the conference heavily depends on the quality of reviews that their submissions receive.
The typical procedure of assigning papers to reviewers is as follows. First, for each paper-reviewer pair, a similarity score is calculated based on various parameters such as the subject area of the paper and the reviewer, the bids placed by the reviewer, etc. [1, 5-8]. Then, an assignment is calculated through an optimization problem, where the usual objectives are to maximize either the utilitarian social welfare, which is the total similarity score of all matched paper-reviewer pairs, or the egalitarian social welfare, which is the least total score of reviewers assigned to any submission. Relevant constraints are imposed to ensure that each submission receives an appropriate number of reviews, reviewer workloads are respected, and any conflicts of interest are avoided.
Peng et al. [9] recently mentioned that a major problem with the prestigious mega conferences is that they constitute the main venues for several communities, and as a result, in some cases, people are asked to review submissions that are beyond their main areas of work. They claim that a reasonable solution is to move to a de-centralized publication process by creating more specialized conferences appropriate for different communities. While specialized conferences definitely have their advantages, the maintenance of large conferences that attract multiple communities is also crucial for the emergence of interdisciplinary ideas that can be reviewed by diverse subject experts. Therefore, it is important to ensure that no group has an incentive to break off due to a feeling of being mistreated by the reviewing procedure of a large conference. In this work, we ask whether it
<sup>1</sup>https://cmt3.research.microsoft.com/
$^{2}$ https://github.com/openreview/openreview-matcher
is possible to modify the existing reviewing processes to resolve this issue by treating the various communities satisfactorily. We clarify that the goal is not to build roadblocks to the creation of specialized conferences, but rather to mitigate the harm imposed on communities in large conferences.
To answer this, we look toward the literature on algorithmic fairness. Specifically, we adapt the group fairness notion known as the core [10]; to the best of our knowledge, we are the first to introduce it to the peer review setting. For a reviewing assignment to be in the core, it must ensure that no community (subset of researchers) can "deviate" by setting up its own conference in which (a) no author reviews her own submission, (b) each submission from within the community is reviewed by just as many reviewers as before, but now from within the community, (c) each reviewer reviews no more papers than before, and (d) the submissions are assigned to better reviewers, making the community happier. Intuitively, this is a notion of group fairness because it ensures that the treatment provided to every group of participants meets their "entitlement" (which is defined by what the group can achieve on its own). It is also a notion of stability (often also known as core stability) because it provides no incentives for any community to break off and get isolated by setting up their own conference instead. Note that this definition provides fairness to every possible community, and not only to predefined groups, as is the case for fairness definitions such as demographic parity and equalized odds that are popular in the machine learning literature [11]. In particular, it ensures fair treatment to emerging interdisciplinary communities even before they become visible.
We consider a simple peer review model in which each agent submits (as the sole author) a number of papers to a conference and also serves as a potential reviewer. A reviewing assignment is valid if each paper is reviewed by  $k_{p}$  reviewers, each reviewer reviews no more than  $k_{a}$  papers, and no agent reviews her own submissions. To ensure that a valid assignment always exists, we assume that the maximum number of papers that each agent is allowed to submit is at most  $\lfloor k_{a} / k_{p} \rfloor$ .
In Section 3, we present an efficient algorithm that always returns a valid assignment in the core under minor conditions on the preferences of the authors. Specifically, our algorithm takes as input only the preference ranking of each author over individual potential reviewers for each of her submissions. Then, it produces an assignment that we prove to be in the core for any manner in which the agent's submission-specific preferences over individual reviewers may be extended to preferences over a set of  $k_{p}$  reviewers assigned to each submission, aggregated over submissions, subject to two mild conditions being satisfied.
In Section 4, we conduct experiments with real data from CVPR and ICLR conferences, and evaluate the price that our algorithm must pay — in lost utilitarian and egalitarian welfare — in order to satisfy the core and prevent communities from having an incentive to deviate. We also observe that reviewer assignment methods currently used in practice generate such adverse incentives quite often.
As we mentioned above, usually the first step of a review assignment procedure is to calculate a similarity score for each pair of submission and reviewer which aims to capture the expertise of the reviewer for this submission. The problem of identifying the similarity scores has been extensively studied in the literature [1, 5-8, 12]. In this work, we assume that the similarity scores are given as an input to our algorithm after they have been calculated from a procedure that is considered as a black box. Importantly, our algorithm does not need the exact values of the similarity scores, but it only requires a ranking of the reviewers for each paper, indicating their relative expertise for this paper.
Given, the similarities scores various methods have been proposed for finding a reviewing assignment. The most famous algorithm is the Toronto Paper Matching System [1] which is a very broadly applied method and focuses on maximizing the utilitarian welfare, i.e. the sum of the similarities across all assigned reviewers and all papers. This approach has been adopted by other popular conference management systems such as EasyChair $^3$  and HotCRP $^4$  [13]. While this approach optimizes the total welfare, it is possible to discriminate against some papers. Therefore, other methods have focused on finding reviewing assignments that are (also) fair across all papers.
3https://easychair.org
4https://hotcrp.com
O'Dell et al. [14] suggest a method where the goal is to maximize the total score that a paper gets, while Stelmakh et al. [13] generalized this method by maximizing the minimum paper score, then maximizing the next smallest paper score, etc. Hartvigsen et al. [15] ensure fairness by requiring that each paper is assigned at least one qualified reviewer. Kobren et al. [16] proposed two algorithms that maximize that total utilitarian under the constraint that each paper should receive a score that exceeds a particular threshold. Payan and Zick [17] used the idea of envy-freeness [18] from the fair division literature to ensure fairness over the submissions. Moreover, some other works have focused on being fair over the reviewers rather than the papers [19, 20]). The core property that we consider in this work can be viewed as a fairness requirement over groups of authors. The reader can find more details about the challenges of the peer review problem in the recent survey of Shah [21].
In our model, the review assignment problem is related to exchange problems with endowments [22], since authors can be viewed as being endowed by their own papers which they wish to exchange with other authors that also serve as reviewers. For the basic exchange problem of housing reallocation, Shapley and Scarf [22] showed that an algorithm called Top-Trading-Cycle (TTC) finds an allocation which is in the core. The first part of our algorithm uses a variation of TTC where the agents (authors) are incorporated with multiple items (submissions), and constraints related to how many items each agent can get and to how many agents one item should be assigned should be satisfied. In contrast to classical exchange problem with endowments, our model has a distinctive requirement that agents/authors need to give away all their items/papers as the papers need to be reviewed by the agent who gets the paper. As we further explain in Section 3, this difference is crucial and requires further action from our algorithm than simply executing this variation of TTC. Various variations of TTC have been considered in the literature, tailored for different variations of the basic problem, but to the best of our knowledge, none of them can be directly applied in our model. To give an example, Suzuki et al. [23] consider the case that there are multiple copies of the same object and there are some quotas that should be satisfied, but they assume that each agent gets just one object while here each paper is assigned to multiple distinct reviewers.
For  $q \in \mathbb{N}$ , define  $[q] \triangleq \{1, \ldots, q\}$ . There is a set of agents  $N = [n]$ . Each agent  $i$  submits a set of papers  $P_i = \{p_{i,1}, \ldots, p_{i,m_i}\}$  for review by her peers, where  $m_i \in \mathbb{N}$ , and is available to review the submissions of her peers. We refer to  $p_{i,\ell}$  as the  $\ell$ -th submission of agent  $i$ ; when considering the special case of each agent  $i$  having a single submission, we will drop  $\ell$  and simply write  $p_i$ . Let  $P = \cup_{i \in N} P_i$  be the set of all submissions and  $m = \sum_{i \in N} m_i$  be the total number of submissions.
Assignment. Our goal is to produce a (reviewing) assignment  $R: N \times P \to \{0,1\}$ , where  $R(i,j) = 1$  if agent  $i \in N$  is assigned to review submission  $j \in P$ . With slight abuse of notation, let  $R_i^a = \{j \in P : R(i,j) = 1\}$  be the set of submissions assigned to agent  $i$  and  $R_j^p = \{i \in N : R(i,j) = 1\}$  be the set of agents assigned to review submission  $j$ . We want the assignment to be valid, i.e., satisfy the following constraints:
Each agent must be assigned at most  $k_{a}$  submissions for review, i.e.,  $|R_{i}^{a}| \leqslant k_{a}, \forall i \in N$ .
Each submission must be assigned to  $k_{p}$  agents, i.e.,  $|R_j^p | = k_p,\forall j\in P$
- No agent should review one of her own submissions, i.e.,  $R(i,p_{i,\ell}) = 0, \forall i \in N, \ell \in [m_i]$ .
To ensure that a valid assignment always exists, we impose the constraint that  $m_i \cdot k_p \leqslant k_a$  for each  $i \in N$ , which implies that  $m \cdot k_p \leqslant n \cdot k_a$ . Intuitively, this demands that each agent submitting papers be willing to provide as many reviews as the number of reviews assigned to the submissions of any single agent. For further discussion on this condition, see Section 5.
Note that given  $N' \subseteq N$  and  $P_i' \subseteq P_i$  for each  $i \in N'$  with  $P' = \cup_{i \in N'} P_i'$ , the validity requirements above can also be extended to a restricted assignment  $\widehat{R}: N' \times P' \to \{0,1\}$ . Hereinafter, we will assume validity unless specified otherwise or during the process of building an assignment.
Preferences. Each agent  $i \in N$  has a preference ranking, denoted  $\sigma_{i,\ell}$ , over the agents in  $N \setminus \{i\}$  for reviewing her  $\ell$ -th submission  $p_{i,\ell}$ . These preferences can be based on a mixture of many factors, such as how qualified the other agents are to review submission  $p_{i,\ell}$ , how likely they are to provide
Our algorithms continue to work with weak orders; one can arbitrarily break ties to convert them into strict orders before feeding them to our algorithms.
a positive review for it, etc. Let  $\sigma_{i,\ell}(i')$  be the position of agent  $i' \in N \setminus \{i\}$  in the ranking. We say that agent  $i$  prefers agent  $i'$  to agent  $i''$  as a reviewer for  $p_{i,\ell}$  if  $\sigma_{i,\ell}(i') < \sigma_{i,\ell}(i'')$ . Again, in the special case where the agents have a single submission each, we drop  $\ell$  and just write  $\sigma_i$ . Let  $\vec{\sigma} = (\sigma_{1,1}, \ldots, \sigma_{1,m_1}, \ldots, \sigma_{n,1}, \ldots, \sigma_{n,m_n})$ .
While our algorithm takes  $\vec{\sigma}$  as input, to reason about its guarantees, we need to define agent preferences over assignments by extending  $\vec{\sigma}$ . In particular, an agent is assigned a set of reviewers for each of her submissions, so we need to define her preferences over sets of sets of reviewers. First, we extend to preferences over sets of reviewers for a given submission, and then aggregate preferences across different submissions. Instead of assuming a specific parametric extension (e.g., additive preferences), we allow all possible extensions that satisfy two mild constraints; the group fairness guarantee of our algorithm holds with respect to any such extension.
Extension to a set of reviewers for one submission: Let  $S \succ_{i,\ell} S'$  (resp.,  $S \succ_{i,\ell} S'$ ) denote that agent  $i$  strictly (resp., weakly) prefers the set of agents  $S$  to the set of agents  $S'$  for her  $\ell$ -th submission  $p_{i,\ell}$ . We require only that these preferences satisfy the following mild axiom.
Definition 1 (Order Separability). For every disjoint  $S_1, S_2, S_3 \subseteq N$  with  $|S_1| = |S_2| > 0$ , if it holds that  $\sigma_{i,\ell}(i') < \sigma_{i,\ell}(i'')$  for each  $i' \in S_1$  and  $i'' \in S_2$ , then we must have  $S_1 \cup S_3 \succ_{i,\ell} S_2 \cup S_3$ .
An equivalent reformulation is that between any two sets of reviewers  $S$  and  $T$  with  $|S| = |T|$ , ignoring the common reviewers in  $S \cap T$ , if the agent strictly prefers every (even the worst) reviewer in  $S \setminus T$  to every (even the best) reviewer in  $T \setminus S$ , then the agent must strictly prefer  $S$  to  $T$ .
Example 1. Consider the common example of additive preferences, where each agent  $i$  has a utility function  $u_{i,\ell}:N\setminus \{i\} \to \mathbb{R}_{\geqslant 0}$  over individual reviewers for her  $\ell$ -th submission, inducing her preference ranking  $\sigma_{i,\ell}$ . In practice, these utilities are sometimes called similarity scores. Her preferences over sets of reviewers are defined via the additive utility function  $u_{i,\ell}(S)\triangleq \sum_{i'\in S}u_{i,\ell}(i')$ . It is easy to check that for any disjoint  $S_1,S_2,S_3$  with  $|S_1| = |S_2| > 0$ ,  $u_{i,\ell}(i') > u_{i,\ell}(i'')$  for all  $i'\in S_1$  and  $i''\in S_2$  would indeed imply  $u_{i,\ell}(S_1\cup S_3) > u_{i,\ell}(S_2\cup S_3)$ . Additive preferences are just one example from a broad class of extensions satisfying order separability.
Extension to assignments. Let us now consider agent preferences over sets of reviews, or equivalently, over assignments. Let  $R \succ_{i} \widehat{R}$  (resp.,  $R \succ_{i} \widehat{R}$ ) denote that agent  $i$  strictly (resp., weakly) prefers assignment  $R$  to assignment  $\widehat{R}$ . Note that these preferences collate the submission-wise preferences  $\succ_{i,\ell}$  across all submissions of the agent. We require only that the preference extension satisfies the following natural property.
Definition 2 (Consistency). For any assignment  $R$ , restricted assignment  $\widehat{R}$  over any  $N' \subseteq N$  and  $P' = \cup_{i \in N'} P_i'$  (where  $P_i' \subseteq P_i$  for each  $i \in N'$ ), and agent  $i^* \in N'$ , if it holds that  $R_{p_{i^*},\ell}^{p} \succ_{i^*,\ell} \widehat{R}_{p_{i^*},\ell}^{p}$  for each  $p_{i^*,\ell} \in P_i'$ , then we must have  $R \succ_{i} \widehat{R}$ .
In words, if an agent weakly prefers  $R$  to  $\widehat{R}$  for the set of reviewers assigned to each of her submissions individually, then she must prefer  $R$  to  $\widehat{R}$  overall.
Example 2. Let us continue with the previous example of additive utility functions. The preferences of agent  $i$  can be extended additively to assignments using the utility function  $u_{i}(R) = \sum_{p_{i,\ell} \in P} u_{i,\ell}(R_{p_{i,\ell}}^{p})$ . It is again easy to check that if  $u_{i,\ell}(R_{p_{i,\ell}}^{p}) \geqslant u_{i,\ell}(\widehat{R}_{p_{i,\ell}}^{p})$  for each  $p_{i,\ell}$ , then  $u_{i}(R) \geqslant u_{i}(\widehat{R})$ . Hence, additive preferences are again one example out of a broad class of preference extensions that satisfy consistency.
Core. Our goal is to find a group-fair assignment which treats every possible group of agents at least as well as they could be on their own, thus ensuring that no subset of agents has an incentive to deviate and set up their own separate conference. Formally:
Definition 3 (Core). An assignment  $R$  is in the core if there is no  $N' \subseteq N$ ,  $P_i' \subseteq P_i$  for each  $i \in N'$ , and restricted assignment  $\widehat{R}$  over  $N'$  and  $P' = \cup_{i \in N'} P_i'$  such that  $\widehat{R} \succ_i R$  for each  $i \in N'$ .
In words, if any subset of agents deviate with any subset of their submissions and implement any restricted reviewing assignment, at least one deviating agent would not be strictly better off, thus eliminating the incentive for such a deviation. We also remark that our algorithm takes only the preference rankings over individual reviewers  $\vec{\sigma}$  as input and produces an assignment  $R$  that is
ALGORITHM 1: CoBRA
Input:  $N,P,\vec{\sigma},k_a,k_p$  Output:  $R$    
1  $R,L,U = \mathrm{PRA - TTC}(N,P,\vec{\sigma},k_a,k_p)$    
2 if  $|U| > 0$  then   
3  $R = Filling-Gaps(N,P,\vec{\sigma},k_a,k_p,R,L,U)$  ALGORITHM 2: PRA-TTC Input:  $N,P,\vec{\sigma},k_a,k_p$  Output:  $R,L,U$    
1  $R(i,j)\gets 0,\forall i\in N$  and  $\forall j\in P$    
2 Construct the preference graph  $G_{R}$    
3 while  $\exists$  cycle in  $G_{R}$  do Eliminate the cycle; Update  $\overline{P}_i$  s by removing any completely assigned paper; Update  $G_{R}$  ..   
7  $U\gets \{i\in N:\overline{P}_i\neq \emptyset \}$  .   
8  $L\gets$  the last  $k_{p} - |U| + 1$  agents in  $N\backslash U$  to have all their submissions completely assigned ;
guaranteed to be in the core according to every preference extension of  $\vec{\sigma}$  satisfying order separability and consistency.
3 CoBRA: An Algorithm for Computing Core-Based Reviewer Assignment
In this section, we prove our main result: when agent preferences are order separable and consistent, an assignment in the core always exists and can be found in polynomial time.
Techniques and key challenges: The main algorithm CoBRA (Core-Based Reviewer Assignment), presented as Algorithm 1, uses two other algorithms, PRA-TTC and Filling-Gaps, presented as Algorithm 2 and Algorithm 3, respectively. We remark that PRA-TTC is an adaptation of the popular Top-Trading-Cycles (TTC) mechanism, which is known to produce an assignment in the core for the house reallocation problem (and its variants) [22]. The adaptation mainly incorporates the constraints related to how many papers each reviewer can review and how many reviewers should review each paper. While for  $k_{p} = k_{a} = 1$ , PRA-TTC is identical with the classic TTC that is used for the house reallocation problem, the main difference of this problem with the review assignment problem is that in the latter each agent should give away her item (i.e. her submission) and obtain the item of another agent. Therefore, by simply executing TTC in the review assignment problem, one can get into a deadlock before producing a valid assignment. For example, consider the case of three agents, each with one submission. Each submission must receive one review ( $k_{p} = 1$ ) and each agent provides one review ( $k_{a} = 1$ ). The TTC mechanism may start by assigning agents 1 and 2 to review each other's submission, but this cannot be extended into a valid assignment because there is no one left to review the submission of agent 3. This is where Filling-Gaps comes in; it makes careful edits to the partial assignment produced by the PRA-TTC, and the key difficulty is to prove that this produces a valid assignment while still satisfying the core.
3.1 Description of CoBRA
Before we describe CoBRA in detail, let us introduce some more notation. Let  $m^{*} = \max_{i\in N}m_{i}$ . For reasons that will become clear later, we want to ensure that  $m_{i} = m^{*}$ , for each  $i\in N$ . To achieve that, we add  $m^{*} - m_{i}$  dummy submissions to agent  $i$ , and the rankings over reviewers with respect to these submissions are arbitrarily. An assignment is called partial if there are submissions that are reviewed by less than  $k_{p}$  agents. A submission that is reviewed by  $k_{p}$  agents under a partial assignment is called completely assigned. Otherwise, it is called incompletely assigned. We denote with  $\overline{P}_i(\widehat{R})$  the set of submissions of  $i$  that are incompletely assigned under a partial assignment  $\widehat{R}$ . We omit  $\widehat{R}$  from the notation when it is clear from context.
CoBRA calls PRA-TTC, and then if needed, it calls Filling-Gaps. Below, we describe the algorithms.
ALGORITHM 3: Filling-Gaps
Input:  $N,P,\vec{\sigma},k_a,k_p,R,L,U$  Output:  $R$  Phase 1;   
1 Construct the greedy graph  $G_{R}$    
2 while  $\exists$  cycle do Eliminate the cycle ; Update  $\overline{P}_i$  -s by removing any completely assigned paper; Update  $U$  and  $L$  by moving any agent  $i$  from  $U$  to  $L$  if  $\bar{P}_i = \emptyset$  Update  $G_{R}$  . Phase 2;   
7 Construct the topological order  $\vec{\rho}$  of  $G_{R}$    
8 for  $t\in [|U|]$  do while  $\overline{P}_{\rho (t)}\neq \emptyset$  do Pick arbitrary  $p_{\rho (t),\ell}\in \overline{P}_{\rho (t)}$  Find completely assigned  $p_{i^{\prime},\ell^{\prime}}$  , with  $R(\rho (t),p_{i^{\prime},\ell^{\prime}}) = 0$  , for some  $i^\prime \in U\cup L\setminus \{\rho (t)\}$  . Find  $i''\neq \rho (t)$  such that  $R(i^{\prime \prime},p_{\rho (t),\ell}) = 0$  and  $R(i^{\prime \prime},p_{i^{\prime},\ell^{\prime}}) = 1$  .  $R(i^{\prime \prime},p_{\rho (t),\ell})\gets 1$  .  $R(i^{\prime \prime},p_{i^{\prime},\ell^{\prime}})\gets 0$  .  $R(\rho (t),p_{i^{\prime},\ell^{\prime}})\gets 1$  Remove  $p_{\rho (t),\ell}$  from  $\overline{P}_{\rho (t)}$  if it is completely assigned;