SlowGuess/NeurIPS_2023_OCR huggingface dataset:
PRA-TTC. In order to define PRA-TTC, we first need to introduce the notion of a preference graph. Suppose we have a partial assignment  $\widehat{R}$ . Each agent  $i$  with  $\overline{P}_i \neq \emptyset$  picks one of her incompletely assigned submissions arbitrarily. Without loss of generality, we assume that she picks her  $\ell^*$ -th submission. We define the directed preference graph  $G_{\widehat{R}} = (N, E_{\widehat{R}})$  where each agent is a node and for each  $i$  with  $\overline{P}_i \neq \emptyset$ ,  $(i, i') \in E_{\widehat{R}}$  if and only if  $i'$  is ranked highest in  $\sigma_{i,\ell^*}$  among the agents that don't review  $p_{i,\ell^*}$  and review less than  $k_a$  submissions. Moreover, for each  $i \in N$  with  $\overline{P}_i = \emptyset$ , we add an edge from  $i$  to  $i'$ , where  $i'$  is an arbitrary agent with  $\overline{P}_{i'} \neq \emptyset$ . PRA-TTC starts with an empty assignment, constructs the preference graph and searches for a directed cycle. If such a cycle exists, the algorithm eliminates it as following: For each  $(i, i')$  that is included in the cycle, it assigns submission  $p_{i,\ell^*}$  to  $i'$  (if  $i$ 's submissions are already completely assigned, it does nothing) and removes  $p_{i,\ell^*}$  from  $\overline{P}_i$ , if it is now completely assigned. Then, the algorithm updates the preference graph and continues to eliminate cycles in the same way. When there are no left cycles in the preference graph, the algorithm terminates and returns two sets,  $U$  and  $L$ . The first set contains all the agents that some of their submissions are incompletely assigned and the set  $L$  contains the last  $k_p - |U| + 1$  agents whose all submissions became completely assigned.
Filling-Gaps. CoBRA calls Filling-Gaps, if the  $U$  that returned from PRA-TTC is non empty. Before we describe the Filling-Gaps, we also need to introduce the notion of a greedy graph. Suppose that we have a partial assignment  $\widehat{R}$  which indicates a set  $U$  that contains all the agents whose at least one submission is incompletely assigned. We define the directed greedy graph  $G_{\widehat{R}} = (U,E_{\widehat{R}})$  where  $(i,i^{\prime})\in E_{\widehat{R}}$  if  $\widehat{R} (i^{\prime},p_{i,\ell}) = 0$  for some  $p_{i,\ell}\in \overline{P}_i$ . In other words, while in the preference graph, agent  $i$  points only to her favourite potential reviewer with respect to one of her incomplete submissions, in the greedy graph agent  $i$  points to any agent in  $U\setminus \{i\}$  that could review at least one of her submissions that is incompletely assigned. Filling-Gaps consists of two phases. In the first phase, starting from the partial assignment  $R$  that was created from PRA-TTC, it constructs the greedy graph, searches for cycles and eliminates a cycle by assigning  $p_{i,\ell}$  to agent  $i^{\prime}$  for each  $(i,i^{\prime})$  in the cycle that exists due to  $p_{i,\ell}$  (when an edge exists due to multiple submissions, the algorithm chooses one of them arbitrary). Then, it updates  $\overline{P}_i$  by removing any  $p_{i,\ell}$  that became completely assigned and also updates  $U$  by moving any  $i$  to  $L$  if  $\overline{P}_i$  became empty. It continues by updating the greedy graph and eliminating cycles in the same way. When no more cycles exist in the greedy graph, the algorithm proceeds to the second phase, where in  $|U|$  rounds ensures that the incomplete submissions of each agent become completely assigned.
In the appendix, there is an execution of CoBRA in a small instance.
3.2 Main Result
We are now ready to present our main result.
Theorem 1. When agent preferences are order separable and consistent, CoBRA returns an assignment in the core in  $O(n^{3})$  time complexity.
Proof. First, in the next lemma, we show that CoBRA returns a valid assignment. The proof is quite non trivial and several pages long, so we defer it to the supplementary material.
Lemma 1. CoBRA returns a valid assignment.
Now, we show that the final assignment  $R$  that CoBRA returns is in the core. Note that while it is possible that an assignment of an agent in  $U \cup L$ , that was established during the execution of PRA-TTC, to be removed in the execution of Filling-Gaps, this never happens for submissions that belong to some agent in  $N \setminus (U \cup L)$ . For the sake of contradiction, assume that  $N' \subseteq N$ , with  $P_i' \subseteq P_i$  for each  $i \in N'$ , deviate to a restricted assignment  $\widetilde{R}$  over  $N'$  and  $\cup_{i \in N'} P_i'$ . Note that  $\widetilde{R}$  is valid only if  $|N'| > k_p$ , as otherwise there is no way each submission in  $\cup_{i \in N'} P_i'$  to be completely assigned, since no agent can review her own submissions.
We distinguish into two cases and we show that in both cases the assignment is in the core.
Case I:  $\exists i \in N': i \notin L \cup U$ . Let  $i^* \in N'$  be the first agent in  $N'$  whose all submissions became completely assigned in the execution of PRA-TTC. Note that since there exists  $i \notin U \cup L$ , we get that  $i^* \notin U \cup L$  from the definitions of  $U$  and  $L$ . Now, consider any  $p_{i^*,\ell}$ . Let  $Q_1 = R_{p_{i^*,\ell}}^p \setminus (R_{p_{i^*,\ell}}^p \cap \widetilde{R}_{p_{i^*,\ell}}^p)$  and  $Q_2 = \widetilde{R}_{p_{i^*,\ell}}^p \setminus (R_{p_{i^*,\ell}}^p \cap \widetilde{R}_{p_{i^*,\ell}}^p)$ . If  $Q_1 = \emptyset$ , then we have that  $R_{p_{i^*,\ell}}^p = \widetilde{R}_{p_{i^*,\ell}}^p$  which means that  $R_{p_{i^*,\ell}}^p \succ_{i^*,\ell} \widetilde{R}_{p_{i^*,\ell}}^p$ . Otherwise, let  $i' = \operatorname{argmax}_{i \in Q_1} \sigma_{i^*,\ell}(i)$ , i.e.  $i'$  is ranked at the lowest position in  $\sigma_{i^*,\ell}$  among the agents that review  $p_{i^*,\ell}$  under  $R$  but not under  $\widetilde{R}$ . Moreover, let  $i'' = \operatorname{argmin}_{i \in Q_2} \sigma_{i^*,\ell}(i)$ , i.e.  $i''$  is ranked at the highest position in  $\sigma_{i^*,\ell}$  among the agents that review  $p_{i^*,\ell}$  under  $\widetilde{R}$  but not under  $R$ . We have  $R(i', p_{i^*,\ell}) = 1$ , if and only if  $i^*$  has an outgoing edge to  $i'$  at some round of PRA-TTC. At the same round, we get that  $i''$  can review more submissions, since  $i'' \in N'$  and if  $i^*$  has incompletely assigned submissions, then any  $i \in N'$  has incompletely assigned submissions, and hence  $|R_{i''}^a| < k_p \cdot m^* \leqslant k_a$ . This means that if  $\sigma_{i^*,\ell}(i') > \sigma_{i^*,\ell}(i'')$ , then  $i^*$  would point  $i''$  instead of  $i'$ . We conclude that  $\sigma_{i^*,\ell}(i') < \sigma_{i^*,\ell}(i'')$ . Then, from the definition of  $i'$  and  $i''$  and from the order separability property we have that  $R_{p_{i^*,\ell}}^p \succ_{i^*,\ell} \widetilde{R}_{p_{i^*,\ell}}^p$ . Thus, either if  $Q_1$  is empty or not, we have that for any  $p_{i^*,\ell} \in P_i'$ , it holds that  $R_{p_{i^*,\ell}}^p \succ_{i^*,\ell} \widetilde{R}_{p_{i^*,\ell}}^p$  and from consistency, we get that  $R \succ_{i^*} \widetilde{R}$  which is a contradiction.
Case II:  $\not\exists i \in N': i \notin L \cup U$ . In this case we have that  $N' = U \cup L$ , as  $|U \cup L| = k_p + 1$ . This means that for each  $i \in U \cup L$  and  $\ell \in [m^*]$ ,  $\widetilde{R}_{p_i,\ell}^p = (U \cup L) \setminus \{i\}$ . Let  $i^* \in L$  be the first agent in  $L$  whose all submissions became completely assigned in the execution of PRA-TTC. Consider any  $p_{i^*,\ell}$ . Note that it is probable that while  $p_{i^*,\ell}$  was assigned to some agent  $i$  in PRA-TTC, it was moved to another agent  $i'$  during the execution of Filling-Gaps. But,  $i'$  belongs to  $U$  and we can conclude that if  $p_{i^*,\ell}$  is assigned to some  $i \in N \setminus U$  at the output of CoBRA, this assignment took place during the execution of PRA-TTC. Now, let  $Q_1 = R_{p_i^*,\ell}^p \setminus (R_{p_i^*,\ell}^p \cap \widetilde{R}_{p_i^*,\ell}^p)$  and  $Q_2 = \widetilde{R}_{p_i^*,\ell}^p \setminus (R_{p_i^*,\ell}^p \cap \widetilde{R}_{p_i^*,\ell}^p)$ . If  $Q_1 = \emptyset$ , then we have that  $R_{p_i^*,\ell}^p = \widetilde{R}_{p_i^*,\ell}^p$  which means that  $R_{p_i^*,\ell}^p \succcurlyeq i^*,\ell \widetilde{R}_{p_i^*,\ell}^p$ . If  $Q_1 \neq \emptyset$ , then  $Q_1 \subseteq N \setminus (U \cup L)$  and  $Q_2 \subseteq U \cup L$  since  $\widetilde{R}_{p_i^*,\ell}^p = U \cup L$ . Let  $i' = \operatorname{argmax}_{i \in Q_1} \sigma_{i^*,\ell}(i)$ , i.e.  $i'$  is ranked at the lowest position in  $\sigma_{i^*,\ell}$  among the agents that review  $p_{i^*,\ell}$  under  $R$  but not under  $\widetilde{R}$ . Moreover, let  $i'' = \operatorname{argmin}_{i \in Q_2} \sigma_{i^*,\ell}(i)$ , i.e.  $i''$  is ranked at the highest position in  $\sigma_{i^*,\ell}$  among the agents that review  $p_{i^*,\ell}$  under  $\widetilde{R}$  but not under  $R$ . From above, we know that the assignment of  $p_{i^*,\ell}$  to  $i'$  was implemented during the execution of PRA-TTC, since  $i' \in N \setminus (U \cup L)$ . Hence, with very similar arguments as in the previous case, we will conclude that  $\sigma_{i^*,\ell}(i') < \sigma_{i^*,\ell}(i'')$ . We have  $R(i', p_{i^*,\ell}) = 1$  if and only if  $i^*$  has an outgoing edge to  $i'$  at some round of PRA-TTC. At this round, we know that  $i''$  can review more submissions, since  $i'' \in N'$  and if  $i^*$  has incompletely assigned submissions, then any  $i \in N'$  has incompletely assigned submissions. This means that if  $\sigma_{i^*,\ell}(i') > \sigma_{i^*,\ell}(i'')$ , then  $i^*$  would point  $i''$  instead of  $i'$ . Hence, we conclude that
$\sigma_{i^{*},\ell}(i^{\prime}) < \sigma_{i^{*},\ell}(i^{\prime \prime})$ . Then, from the definition of  $i^{\prime}$  and  $i^{\prime \prime}$  and from the order separability property we have that  $R_{p_{i^{*},\ell}}^{p} \succ_{i^{*},\ell} \widetilde{R}_{p_{i^{*},\ell}}^{p}$ . Thus, either if  $Q_{1}$  is empty or not, we have that for any  $p_{i^{*},\ell} \in P_{i}^{\prime}$ , it holds that  $R_{p_{i^{*},\ell}}^{p} \succ_{i^{*},\ell} \widetilde{R}_{p_{i^{*},\ell}}^{p}$  and from consistency we get that  $R \succ_{i^{*}} \widetilde{R}$  which is a contradiction.
Lastly, we analyze the time complexity of CoBRA. First, we consider the time complexity of PRA-TTC. In each iteration, the algorithm assigns at least one extra reviewer to at least one incompletely-assigned submission. This can continue for at most  $m \cdot k_p \leqslant n \cdot k_a$  iterations, since each submission should be reviewed by  $k_p$  reviewers. In each iteration, it takes  $O(n)$  time to find and eliminate a cycle in the preference graph. Then, it takes  $O(n^2)$  time to update the preference graph, since for each arbitrarily-picked incompletely-assigned submission of each agent, we need to find the most qualified reviewer who can be additionally assigned to it. By all the above, we conclude that the runtime of PRA-TTC is  $O(n^3)$ , by ignoring  $k_a$  which is a small constant in practice. After PRA-TTC terminates, CoBRA calls the Filling-Gaps algorithm. However, Lemma 3 ensures that at the end of PRA-TTC,  $|L \cup U| \leqslant k_p + 1$ , which is also a small constant. And Filling-Gaps only makes local changes that affect these constantly many agents. As such, the running time of Filling-Gaps is constant as well. Therefore, the time complexity of CoBRA is  $O(n^3)$
In this section, we empirically compare CoBRA to TPMS [1], which is widely used (for example, it was used by NeurIPS for many years), and PR4A [13], which was used in ICML 2020 [24]. As mentioned in the introduction, these algorithms assume the existence of a similarity or affinity score for each pair of reviewer  $i$  and paper  $j$ , denoted by  $S(i,j)$ . The score (or utility) of a paper under an assignment  $R$ , denoted by  $u_{j}^{p}$ , is computed as  $u_{j}^{p} \triangleq \sum_{i \in R_{j}^{p}} S(i,j)$ . TMPS finds an assignment  $R$  that maximizes the utilitarian social welfare (USW), i.e., the total paper score  $\sum_{j \in P} u_{j}^{p}$ , whereas PR4A finds an assignment that maximizes the egalitarian social welfare (ESW), i.e., the minimum paper score  $\min_{j \in P} u_{j}^{p}$ . We use  $k_{a} = k_{p} = 3$  in these experiments.
Datasets. We use three conference datasets: from the Conference on Computer Vision and Pattern Recognition (CVPR) in 2017 and 2018, which were both used by Kobren et al. [16], and from the International Conference on Learning Representations (ICLR) in 2018, which was used by Xu et al. [25]. In the ICLR 2018 dataset, similarity scores and conflicts of interest are also available. While a conflict between a reviewer and a paper does not necessarily indicate authorship, it is the best indication we have available, so, following Xu et al. [25], we use the conflict information to deduce authorship. Since in our model each submission has one author, and no author can submit more than  $\lfloor k_a / k_p\rfloor = 1$  papers, we compute a maximum cardinality matching on the conflict matrix to find author-paper pairs, similarly to what Dhull et al. [26] did. In this way, we were able to match 883 out of the 911 papers. We disregard any reviewer who does not author any submissions, but note that the addition of more reviewers can only improve the results of our algorithm since these additional reviewers have no incentive to deviate. For the CVPR 2017 and CVPR 2018 datasets, similarity scores was available, but not the conflict information. In both these datasets, there are fewer reviewers than papers. Thus, we constructed artificial authorship relations by sequentially processing papers and matching each paper to the reviewer with the highest score for it, if this reviewer is still unmatched. In this way, we were able to match 1373 out of 2623 papers from CVPR 2017 and 2840 out of 5062 papers from CVPR 2018. In the ICLR 2018 and CVPR 2017 datasets, the similarity scores take values in  $[0,1]$ , so we accordingly normalized the CVPR 2018 scores as well.
Measures. We are most interested in measuring the extent to which the existing algorithms provide incentives for communities of researchers to deviate. To quantify this, we need to specify the utilities of the authors. We assume that they are additive, i.e., the utility of each author in an assignment is the total similarity score of the  $k_{p} = 3$  reviewers assigned to their submission.
Technically, subject to this, it maximizes the second minimum paper score, and then the third minimum paper score, etc. This refinement is also known as leximin in the literature.
Note that  $k_{p} = 3$  reviews per submission is quite common, although the reviewer load  $k_{a}$  is typically higher in many conferences, often closer to 6. However, the differences between different algorithms diminish with higher values of  $k_{a}$ .
Dataset	Alg	USW	ESW	α-Core	CV-Pr
#unb-α	α*
CVPR '17	CoBRA	1.225 ± 0.021	0.000 ± 0.000	0%	1.000 ± 0.000	0%
TPMS	1.497 ± 0.019	0.000 ± 0.000	89%	3.134 ± 0.306	100%
PR4A	1.416 ± 0.019	0.120 ± 0.032	51%	1.700 ± 0.078	100%
CVPR '18	CoBRA	0.224 ± 0.004	0.004 ± 0.001	0%	1.000 ± 0.000	0%
TPMS	0.286 ± 0.005	0.043 ± 0.004	0%	1.271 ± 0.038	100%
PR4A	0.282 ± 0.005	0.099 ± 0.001	0%	1.139 ± 0.011	100%
ICLR '18	CoBRA	0.166 ± 0.001	0.028 ± 0.001	0%	1.000 ± 0.000	0%
TPMS	0.184 ± 0.001	0.048 ± 0.002	0%	1.048 ± 0.008	90%
PR4A	0.179 ± 0.001	0.082 ± 0.001	0%	1.087 ± 0.009	100%
Table 1: Results on CVPR 2017 and 2018, and ICLR 2018.
Core violation factor: Following the literature [27], we measure the multiplicative violation of the core (if any) that is incurred by TPMS and PR4A (CoBRA provably does not incur any). This is done by computing the maximum value of  $\alpha \geqslant 1$  for which there exists a subset of authors such that by deviating and implementing some valid reviewing assignment of their papers among themselves, they can each improve their utility by a factor of at least  $\alpha$ . This can easily be formulated as a binary integer linear program (BILP). Because this optimization is computationally expensive (the most time-consuming component of our experiments), we subsample 100 papers from each dataset in each run, and report results averaged over 100 runs. Note that whenever there exists a subset of authors with zero utility each in the current assignment who can deviate and receive a positive utility each, the core deviation  $\alpha$  becomes infinite. We separately measure the percentage of runs in which this happens (in the column #unb- $\alpha$ ), and report the average  $\alpha$  among the remaining runs in the  $\alpha^{*}$  column.
Core violation probability: We also report the percentage of runs in which a core violation exists (i.e., there exists at least one subset of authors who can all strictly improve by deviating from the current assignment). We refer to this as the core violation probability (CV-Pr).
Social welfare: Finally, we also measure the utilitarian and egalitarian social welfare (USW and ESW) defined above, which are the objectives maximized by TPMS and PR4A, respectively.
Results. The results are presented in Table 1. As expected, TPMS and PR4A achieve the highest USW and ESW, respectively, on all datasets because they are designed to optimize these objectives. In CVPR 2017, CoBRA and TPMS always end up with zero ESW because this dataset includes many zero similarity scores, but PR4A is able to achieve positive ESW. In all datasets, CoBRA achieves a relatively good approximation with respect to USW, but this is not always the case with respect to ESW. For example, in CVPR 2018, CoBRA achieves 0.004 ESW on average whereas PR4A achieves 0.099 ESW on average. This may be due to the fact that this dataset also contains many zero similarity scores, and the myopic process of CoBRA locks itself into a bad assignment, which the global optimization performed by PR4A avoids.
While CoBRA suffers some loss in welfare, TPMS and PR4A also generate significant adverse incentives. They incentivize at least one community to deviate in almost every run of each dataset (CV-Pr). While the magnitude of this violation is relatively small when it is finite (except for TPMS in CVPR 2017), TPMS and PR4A also suffer from unbounded core violations in more than half of the runs for CVPR 2017; this may again be due to the fact that many zero similarity scores lead to deviations by groups where each agent has zero utility under the assignments produced by these algorithms.
Of all these results, the high probability of core violation under TPMS and PR4A is perhaps the most shocking result; when communities regularly face adverse incentives, occasional deviations may happen, which can endanger the stability of the conference. That said, CoBRA resolves this issue at a significant loss in fairness (measured by ESW). This points to the need for finding a middle ground where adverse incentives can be minimized without significant loss in fairness or welfare.
In Table 2 in the appendix, we report USW and ESW without any subsampling and we note that the qualitative relationships between the different algorithms according to each metric remain the same.
In this work, we propose a way for tackling the poor reviewing problem in large conferences by introducing the concept of "core" as a notion of group fairness in the peer review process. This fairness principle ensures that each subcommunity is treated at least as well as it would be if it was not part of the larger conference community.
We show that under certain — albeit sometimes unrealistic— assumptions, a peer review assignment in the core always exists and can be efficiently found. In the experimental part, we provide evidence that peer review assignment procedures that are currently used in practice, quite often motivate subcommunities to deviate and build their own conferences.
Our theoretical results serve merely as the first step toward using it to find reviewer assignments that treat communities fairly and prevent them from deviating. As such, our algorithm has significant limitations that must be countered before it is ready for deployment in practice. A key limitation is that it only works for single-author submissions, which may be somewhat more realistic for peer review of grant proposals, but unrealistic for computer science conferences. We also assume that each author serves as a potential reviewer; while many conferences require this nowadays, exceptions must be allowed in special circumstances. We also limit the number of submissions by any author to be at most  $\lfloor k_a / k_p\rfloor$  , which is a rather small value in practice, and some authors ought to submit more papers than this. We need to make this assumption to theoretically guarantee that a valid assignment exists. An interesting direction is to design an algorithm that can produce a valid assignment in the (approximate) core whenever it exists. Finally, deploying group fairness in real-world peer review processes may require designing algorithms that satisfy it approximately at minimal loss in welfare, as indicated by our experimental results.
[1] Laurent Charlin and Richard Zemel. The toronto paper matching system: an automated paper-reviewer assignment system. In Proceedings of the ICML Workshop on Peer Reviewing and Publishing Models, 2013.
[2] G. David L. Travis and Harry M. Collins. New light on old boys: Cognitive and institutional particularism in the peer review system. Science, Technology, & Human Values, 16(3):322-341, 1991.
[3] Raymond S. Nickerson. What authors want from journal reviewers and editors. American Psychological, pages 661--662, 2005.
[4] Nihar B. Shah. Challenges, experiments, and computational solutions in peer review. Commun. ACM, 65(6):76-87, 2022.
[5] David Mimno and Andrew McCallum. Expertise modeling for matching papers with reviewers. In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 500-509, 2007.
[6] Xiang Liu, Torsten Suel, and Nasir Memon. A robust model for paper reviewer assignment. In Proceedings of the 8th ACM Conference on Recommender systems, pages 25-32, 2014.
[7] Marko A. Rodriguez and Johan Bollen. An algorithm to determine peer-reviewers. In Proceedings of the 17th ACM conference on Information and knowledge management, pages 319-328, 2008.
[8] Hong Diep Tran, Guillaume Cabanac, and Gilles Hubert. Expert suggestion for conference program committees. In 2017 11th International Conference on Research Challenges in Information Science (RCIS), pages 221-232. IEEE, 2017.
[9] Andi Peng, Jessica Zosa Forde, Yonadav Shavit, and Jonathan Frankle. Strengthening subcommunities: Towards sustainable growth in ai research. arXiv preprint arXiv:2204.08377, 2022.
[10] Donald Bruce Gillies. Some theorems on  $n$ -person games. Princeton University, 1953.