SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For a given parameter  $Q \in [0,1]$ , a perturbation function  $f$  and a precision  $w \in \mathbb{N}^{+}$ :
(1) Construct a graph  $G$  with a source  $s$ , a sink  $t$ .
(2) For paper  $p$ , add a vertex  $v_{p}$  and an edge  $s \to v_{p}$  with capacity  $\ell_{p} \cdot w$  and cost 0.
(3) For reviewer  $r$ , add a vertex  $v_{r}$  and an edge  $v_{r} \to t$  with capacity  $\ell_r \cdot w$  and cost 0.
(4) For a reviewer-paper pair  $(p,r)$ , add  $\lfloor Q \cdot w \rfloor$  edges  $v_{p} \to v_{r}$ . The  $i$ -th edge has capacity 1 and cost  $S_{p,r} \cdot \left[f\left(\frac{i}{w}\right) - f\left(\frac{i - 1}{w}\right)\right]$ . Let this set of edges be  $E_{p,r}$ .
(5) Run maximum cost maximum flow algorithm [37] on  $G$ . Define the assignment  $x_{p,r}$  such that  $x_{p,r} = [\text{The total flow on edges in } E_{p,r}] / w$ .
(6) Enumerate all pairs  $(p,r)$  in any order and increase  $x_{p,r}$  by  $\min \{Q - x_{p,r},\ell_p-\sum_r x_{p,r},\ell_r - \sum_p x_{p,r}\}$ . If  $\sum_{p,r}x_{p,r} = n_p\cdot \ell_p$ , return  $\mathbf{x}$ . Otherwise, return infeasible.
At a high level, Algorithm 1 uses a piecewise linear function with  $w$  pieces to approximate the concave function  $f(x)$  and solves the approximated objective with maximum cost maximum flow. As we increase the precision  $w$ , the approximation becomes more accurate, but the running time of the algorithm also scales up. Formally, we have the following Theorem 1.
Theorem 1. Let  $w \in \mathbb{N}^{+}$  be the precision in Algorithm 1, and OPT be the optimal perturbed quality.
(a) Algorithm 1 runs in  $O(w \cdot \ell_p \cdot n_p^2 \cdot n_r)$  time.
(b) Algorithm 1 returns an assignment with perturbed quality  $\mathrm{ALG} \geq \mathrm{OPT} - f\left(\frac{1}{w}\right) \sum_{p,r} S_{p,r}$ .
The proof of Theorem 1 is deferred to Appendix B.1.
Consider the running time of Algorithm 1 given in Theorem 1 (a). If we directly model PM as an optimization problem, the number of variables will be  $n = n_p \cdot n_r$ . The state-of-the-art algorithm for solving a linear program with  $n$  variables to high accuracy has a time complexity of  $O^*(n^{2.38} \log n)$  [38]. In contrast, for fixed  $w$ , Algorithm 1 works in  $O(n^2)$  time since  $\ell_p \leq n_r$ . Thus, Algorithm 1 has a better time complexity than directly solving PM as an optimization program even if the objective is linear. For non-linear perturbation functions, the time complexity of Algorithm 1 remains the same while the complexity of solving the optimization program increases. Moreover, by Theorem 1 (b), we can see that as  $w$  increases, the approximated perturbed quality ALG approaches OPT, formalizing the intuition that as we increase the precision  $w$ , the approximation becomes more accurate. In Section 6, we empirically evaluate the running time and the approximation accuracy of Algorithm 1. We find with  $w = 10$ , Algorithm 1 produces decently-accurate approximations on our datasets.
5 Theoretical Analysis
In this section, we provide two theorems showing that PM provably outperforms PLRA on a general class of input instances. We start with the simpler one inspired by the example in Fig. 1. Let  $\mathrm{PLRA}(Q)$  and  $\mathrm{PM}(Q, f)$  be the set of possible solutions of PLRA and PM respectively.
Definition 5.1 (Blockwise Dominant Matrix). A similarity matrix  $\mathbf{S} \in \mathbb{R}_{\geq 0}^{n_p \times n_r}$  is blockwise dominant with block identity  $\mathbf{A} \in \mathbb{R}_{\geq 0}^{k \times k}$  and block sizes  $\{p_1, \ldots, p_k\}, \{r_1, \ldots, r_k\}$  if
$$
\mathbf {S} = \left( \begin{array}{c c c c} A _ {1, 1} \cdot \mathbf {1} _ {p _ {1} \times r _ {1}} & A _ {1, 2} \cdot \mathbf {1} _ {p _ {1} \times r _ {2}} & \ldots & A _ {1, k} \cdot \mathbf {1} _ {p _ {1} \times r _ {k}} \\ A _ {2, 1} \cdot \mathbf {1} _ {p _ {2} \times r _ {1}} & A _ {2, 2} \cdot \mathbf {1} _ {p _ {2} \times r _ {2}} & \ldots & A _ {2, k} \cdot \mathbf {1} _ {p _ {2} \times r _ {k}} \\ \ldots & \ldots & \ldots & \ldots \\ A _ {k, 1} \cdot \mathbf {1} _ {p _ {k} \times r _ {1}} & A _ {k, 2} \cdot \mathbf {1} _ {p _ {k} \times r _ {2}} & \ldots & A _ {k, k} \cdot \mathbf {1} _ {p _ {k} \times r _ {k}} \end{array} \right),
$$
where  $A_{i,i} > A_{i,j}, \forall i \neq j$  and  $\mathbf{1}_{p \times r}$  is a  $p \times r$  matrix with all entries being 1. Moreover, define the dominance factor of  $\mathbf{A}$  as  $\operatorname{Dom}(\mathbf{A}) = \sup \{\alpha \mid A_{i,i} \geq \alpha \cdot A_{i,j}, \forall i \neq j\}$ .
Remark. Like Fig. 1, a blockwise dominant similarity matrix models a conference with  $k$  subject areas where the  $i$ -th subject area has  $p_i$  papers and  $r_i$  reviewers. The similarity between a paper and a reviewer is determined only by their subject areas, and a paper has highest similarity with a reviewer in the same subject area. Note that  $\mathrm{Dom}(\mathbf{A}) > 1$  as  $A_{i,i} > A_{i,j}, \forall i \neq j$ .
Theorem 2. For an input instance  $(n_p, n_r, \ell_p, \ell_r, \mathbf{S})$ , where  $\mathbf{S}$  is blockwise dominant with block identity  $\mathbf{A} \in \mathbb{R}_{\geq 0}^{k \times k}$  and block sizes  $\{p_1, \ldots, p_k\}, \{r_1, \ldots, r_k\}$ , assume (i)  $\{r_1, \ldots, r_k\}$  are not all equal, (ii)  $p_i \cdot \ell_p \leq r_i \cdot \ell_r \forall i \in \{1, \ldots, k\}$  and (iii)  $Q \cdot r_i \geq \ell_p \forall i \in \{1, \ldots, k\}$ . Let  $f$  be a strictly concave perturbation function and  $f'(0) < \operatorname{Dom}(\mathbf{A}) f'(1)$ . PM with  $f(x)$  as the perturbation function (weakly) dominates PLRA in quality and all randomness metrics. Formally,
(a)  $\forall \mathbf{x}\in \mathrm{PM}(Q,f),\forall \mathbf{y}\in \mathrm{PLRA}(Q),$
$\mathrm{Quality}(\mathbf{x}) \geq \mathrm{Quality}(\mathbf{y}), \quad \mathrm{Maxprob}(\mathbf{x}) \leq \mathrm{Maxprob}(\mathbf{y}), \quad \mathrm{AvgMaxp}(\mathbf{x}) \leq \mathrm{AvgMaxp}(\mathbf{y}),$  L2Norm(x)  $\leq$  L2Norm(y), Entropy(x)  $\geq$  Entropy(y), Support(x)  $\geq$  Support(y).
(b)  $\forall \mathbf{x}\in \mathrm{PM}(Q,f),\exists \mathbf{y}\in \mathrm{PLRA}(Q),$
$\operatorname{AvgMaxp}(\mathbf{x}) < \operatorname{AvgMaxp}(\mathbf{y}),$  L2Norm(x)  $<  \mathrm{L2Norm}(\mathbf{y}),$  Entropy(x) > Entropy(y).
The proof of Theorem 2 follows the same intuition as the example in Fig. 1. The assumptions guarantee that an assignment like Fig. 1a, i.e., uniformly matching papers to reviewers within the same subject area, is feasible. Details of the proof are deferred to Appendix B.2. At a high level, Theorem 2 shows that with a slight restriction on the perturbation function, PM provably performs better than PLRA on blockwise dominant similarity matrices.
Theorem 2 requires a strict restriction on  $\mathbf{S}$ 's structure. In our next theorem, we remove the restriction, stating that PM also outperforms PLRA on a random similarity matrix with high probability.
Theorem 3. For an input instance  $(n_p, n_r, \ell_p, \ell_r, \mathbf{S})$ , where each entry of  $\mathbf{S}$  is i.i.d. sampled from  $\{v_1, \ldots, v_k\} (0 < v_1 < \dots < v_k)$  uniformly, assume (i)  $k \leq \frac{1}{c} \cdot n_p$ , (ii)  $\ell_r \geq c \cdot \ln(n_r)$ , (iii)  $2 \cdot n_p \cdot \ell_p \leq n_r \cdot \ell_r$  and (iv)  $Q \cdot (n_r - 1) \geq \ell_p$ . Let  $f$  be a strictly concave perturbation function and  $f'(0) < \frac{v_i}{v_{i-1}} f'(1), \forall i \in \{2, \ldots, k\}$ . With probability  $1 - e^{-\Omega(c)}$ , PM with  $f(x)$  as the perturbation function (weakly) dominates PLRA in quality and all randomness metrics. Formally,
$\mathrm{Support}(\mathbf{x}) > \mathrm{Support}(\mathbf{y}),$  L2Norm(x)  $<  \mathrm{L2Norm}(\mathbf{y}),$  Entropy(x)  $>$  Entropy(y).
The complete proof of Theorem 3 is deferred to Appendix B.3. To sketch the proof, we will first relate PLRA and PM to two simpler auxiliary algorithms using a concentration inequality and then
prove the dominance between them. The assumptions (ii) and (iii) are used to connect PM and PLRA with the auxiliary algorithms, while (i) and (iv) are for proving the dominance.
For Theorem 3 to hold on general random similarity matrices, we have made a major assumption (i), which requires that there are not too many distinct levels of similarity scores. Assumption (i) is naturally satisfied in some cases. For example, when the similarities are derived purely from reviewers' bids, the number of levels becomes the number of discrete bid levels; discrete values can also result when computing similarities from the overlap between reviewer- and author-selected subject areas. When the similarity scores are instead computed from continuous values like TPMS scores [1], assumption (i) is usually not satisfied. In Section 6, we show experiments on both discrete and continuous similarities to evaluate the effect of assumption (i).
Datasets. In this section, we test our algorithm on two realistic datasets. The first dataset is bidding data from the AAMAS 2015 conference [39]. In this dataset,  $n_p = 613$ ,  $n_r = 201$ , and the bidding data has 4 discrete levels: "yes", "maybe", "no" and "conflict". We transform these 4 levels to similarities  $1, \frac{1}{2}, \frac{1}{4}$  and  $0$ , as inspired by NeurIPS 2016, in which similarity scores were computed as  $2^{\mathrm{bid}}(0.5s_{\mathrm{text}} + 0.5s_{\mathrm{subject}})$  [19]. The transformed dataset satisfies assumption (i) in Theorem 3 as there are only 4 levels. The second dataset contains text-similarity scores recreated from the ICLR 2018 conference with  $n_p = 911$ ,  $n_r = 2435$  [26]. These scores were computed by comparing the text of each paper with the text of each reviewer's past work; we directly use them as the similarity matrix. Assumption (i) is not valid in this dataset as it is continuous-valued. The constraints are set as  $\ell_p = 3$ ,  $\ell_r = 6$  for ICLR 2018 as was done in [26] and  $\ell_p = 3$ ,  $\ell_r = 12$  for AAMAS 2015 for feasibility. In Appendix A.3, we also test our algorithm on four additional datasets from [39].
Experiment and hyperparameter setting. We implement PLRA and two versions of PM (PM-E and PM-Q) using commercial optimization solver Gurobi 10.0 [36]. For each algorithm on each dataset, we use a principled method (Appendix A.2) to find 8 sets of hyperparameters that produce solutions with at least  $\{80\%, 85\%, 90\%, 95\%, 98\%, 99\%, 99.5\%, 100\}$  of the maximum possible quality. When setting the hyperparameters, we choose a "slackness" value  $\delta$  and allow PM to produce solutions with Maxprob at most  $\delta$  higher than the optimal Maxprob (in exchange for better performance on other metrics). For AAMAS 2015,  $\delta = 0.00$  and for ICLR 2018,  $\delta = 0.02$ . We refer to Appendix A.2 for more details. We do not consider assignments with lower than  $80\%$  of the maximum quality since such low-quality assignments are unlikely to be deployed in practice. We evaluate the produced assignments on different randomness metrics to draw Figs. 2 to 4. We also implemented Algorithm 1 and tested it with the set of parameters at  $95\%$  relative quality to create Table 1. All source code is released at https://github.com/YixuanEvenXu/perturbed-maximization, and all experiments are done on a server with 56 cores and 504G RAM, running Ubuntu 20.04.6.
Implementation	Gurobi	Flow (w = 10)
Algorithm	PM-Q	PM-E	PM-Q	PM-E
Wall Clock Time	86.85s	368.13s	563.24s	554.07s
Total CPU Time	988.11s	3212.18s	562.77s	553.59s
Quality	95%	95%	94%	94%
PQuality	606.55	305.41	603.34	303.68
AvgMaxp (↓)	0.87	0.87	0.86	0.86
Entropy (↑)	1589.95	1630.80	1575.86	1607.20
Table 1: Comparison of different implementations on ICLR 2018. Downarrows  $(\downarrow)$  mean the lower the better and uparrows  $(\uparrow)$  mean the higher the better. Due to parallelization, Gurobi takes less wall clock time. However, the network-flow-based approximation uses fewer computation resources (total CPU time). The approximation has relatively accurate Quality, PQuality and randomness metrics.
Comparing Gurobi and Algorithm 1. Table 1 shows that on ICLR 2018, Gurobi takes less wall clock time than Algorithm 1 due to parallelization over the server's 112 cores. However, Algorithm 1 takes less total CPU time (system + user CPU time) and achieves similar performance with Gurobi, which shows that Algorithm 1 provides decently approximated solutions using fewer computation
resources. Due to the space limit, rows in Table 1 have been selected and only results on ICLR 2018 are present. The complete tables and analysis for both datasets can be found in Appendix A.1.
Comparing randomness metrics of PM and PLRA on discrete-valued datasets. As shown in Figs. 2a and 3, on AAMAS 2015, both versions of PM achieve exactly the same performance with PLRA on Maxprob while improving significantly on the other randomness metrics. Recall that PLRA achieves the optimal quality for a given Maxprob. This demonstrates that as predicted by Theorem 3, PM produces solutions that are more generally random while preserving the optimality in Maxprob on discrete-valued datasets where assumption (i) holds. In Appendix A.3, we also test our algorithm on four more datasets from [39] and observe similar results.
(a) AAMAS 2015 Quality vs Maxprob  $(\downarrow)$
(b) ICLR 2018 Quality vs Maxprob  $(\downarrow)$
Figure 2: The trade-offs between quality and Maxprob on both datasets. Downarrows  $(\downarrow)$  indicate that lower is better. On AAMAS 2015 where assumption (i) of Theorem 3 holds, PM incurs no increase in Maxprob; on ICLR 2018 where it does not hold, PM causes a 0.02 increase.
(a) AAMAS 2015 Quality vs AvgMaxp  $(\downarrow)$
(c) AAMAS 2015 Quality vs Support  $(\uparrow)$
Figure 3: The trade-offs between quality and four randomness metrics on AAMAS 2015. Downarrows  $(\downarrow)$  indicate that lower is better and uparrows  $(\uparrow)$  indicate that higher is better. The figure shows that PM significantly outperforms PLRA on all four metrics, as predicted by Theorem 3.
(b) AAMAS 2015 Quality vs L2Norm  $(\downarrow)$
(d) AAMAS 2015 Quality vs Entropy  $(\uparrow)$
Comparing randomness metrics of PM and PLRA on continuous-valued datasets. The results on ICLR 2018 in Figs. 2b and 4 show that both versions of PM sacrifice some Maxprob as compared to PLRA. The exact amount of this sacrifice is affected by the slackness  $\delta$  described in Appendix A.2. However, the improved randomness is still significant, though to a lesser extent than on AAMAS 2015. Note that assumption (i) of Theorem 3 does not hold on ICLR 2018. This exhibits the empirical efficacy of PM beyond the theoretical guarantees provided in Section 5.
Comparing randomness metrics of PM-E and PM-Q. In all of Figs. 2 to 4, we can see that no matter what metric and dataset are used, the performances of PM-E and PM-Q are always similar.
(a) ICLR 2018 Quality vs AvgMaxp  $(\downarrow)$
(c) ICLR 2018 Quality vs Support (↑)
Figure 4: The trade-offs between quality and four randomness metrics on ICLR 2018. Downarrows  $(\downarrow)$  indicate that lower is better and uparrows  $(\uparrow)$  indicate that higher is better. The figure shows that PM outperforms PLRA on all four metrics despite the fact that assumption (i) of Theorem 3 does not hold, indicating the empirical efficacy of PM beyond theoretical guarantees.
(b) ICLR 2018 Quality vs L2Norm  $(\downarrow)$
(d) ICLR 2018 Quality vs Entropy (↑)
This suggests that the specific type of the perturbation function has limited impact as long as it is strictly concave. Therefore, program chairs do not need to carefully consider this choice in practice.
Additional experiments. Due to the page limit on the main paper, we are not able to present all the experiments we run in this section. We refer to Appendix A for additional experimental results about the network-flow-based approximation (Appendix A.1), hyperparameter tuning (Appendix A.2) and experiments on more datasets (Appendix A.3).
We present a general-purpose, practical algorithm for use by conference program chairs in computing randomized paper assignments. We show both theoretically and experimentally that our algorithm significantly improves over the previously deployed randomized assignment algorithm, PLRA. In conferences that currently deploy PLRA, our algorithm can be simply plugged-in in place of it to find a paper assignment with the same quality but with an improved level of randomization, achieving various benefits.
In practice, various aspects of paper assignment other than quality and randomness are considered by program chairs. We refer to Appendix C for discussions about how to optimize a specific metric with our algorithm (Appendix C.1) and how to incorporate some additional constraints from [20] (Appendix C.2). That being said, we do not consider various other aspects and constraints that may be desired by program chairs: e.g., fairness-based objectives [2] or constraints on review cycles [20]. Incorporating these aspects remains an interesting direction for future work.
Broader Impacts. We expect our algorithm to have a mostly positive social impact by assisting program chairs in mitigating malicious behavior, facilitating evaluation of assignments, and increasing reviewer diversity and reviewer anonymity. While deploying a randomized assignment may negatively impact the assignment quality as compared to a deterministic assignment, our work allows conferences to choose the level of randomization that they deem best for their goals.
This work was supported by ONR grant N000142212181 and NSF grant IIS-2200410.
This work was supported in part by the Sloan Research Fellowship.
[1] Laurent Charlin and Richard S. Zemel. The Toronto Paper Matching System: An automated paper-reviewer assignment system. In ICML Workshop on Peer Reviewing and Publishing Models, 2013.
[2] Ivan Stelmakh, Nihar B. Shah, and Aarti Singh. PeerReview4All: Fair and accurate reviewer assignment in peer review. In ALT, 2019.
[3] Steven Jecmen, Hanrui Zhang, Ryan Liu, Fei Fang, Vincent Conitzer, and Nihar B Shah. Near-optimal reviewer splitting in two-phase paper reviewing and conference experiment design. In HCOMP, 2022.
[4] Wenbin Tang, Jie Tang, and Chenhao Tan. Expertise matching via constraint-based optimization. In International Conference on Web Intelligence and Intelligent Agent Technology, 2010.
[5] Peter A. Flach, Sebastian Spiegler, Bruno Golénia, Simon Price, John Guiver, Ralf Herbrich, Thore Graepel, and Mohammed J. Zaki. Novel tools to streamline the conference review process: Experiences from SIGKDD'09. SIGKDD Explorations Newsletter, 2010.
[6] Camillo J. Taylor. On the optimal assignment of conference papers to reviewers. Technical report, Department of Computer and Information Science, University of Pennsylvania, 2008.
[7] Laurent Charlin, Richard S. Zemel, and Craig Boutilier. A framework for optimizing paper matching. In UAI, 2011.
[8] Nihar B Shah. Challenges, experiments, and computational solutions in peer review. Communications of the ACM, 2022.
[9] David Mimno and Andrew McCallum. Expertise modeling for matching papers with reviewers. In KDD, 2007.
[10] Xiang Liu, Torsten Suel, and Nasir Memon. A robust model for paper reviewer assignment. In RecSys, 2014.
[11] Marko A. Rodriguez and Johan Bollen. An algorithm to determine peer-reviewers. In CIKM, 2008.
[12] Hong Diep Tran, Guillaume Cabanac, and Gilles Hubert. Expert suggestion for conference program committees. In RCIS, 2017.
[13] Judy Goldsmith and Robert H. Sloan. The AI conference paper assignment problem. In AAAI Workshop, 2007.
[14] T. N. Vijaykumar. Potential organized fraud in ACM/IEEE computer architecture conferences. https://medium.com/@tnvijayk/potential-organized-fraud-in-acm-ieee-computer-architecture-conferences-ccd61169370d, 2020. Accessed February 1, 2023.
[15] Michael Littman. Collusion rings threaten the integrity of computer science research. Communications of the ACM, 2021.
[16] Jef Akst. I hate your paper. Many say the peer review system is broken. Here's how some journals are trying to fix it. The Scientist, 2010.
[17] Edward F. Barroga. Safeguarding the integrity of science communication by restraining 'rational cheating' in peer review. Journal of Korean Medical Science, 2014.
[18] Mario Paolucci and Francisco Grimaldo. Mechanism change in a simulation of peer review: From junk support to elitism. Scientometrics, 2014.
[19] Nihar B Shah, Behzad Tabibian, Krikamol Muandet, Isabelle Guyon, and Ulrike Von Luxburg. Design and analysis of the nips 2016 review process. Journal of Machine Learning Research, 2018.
[20] Kevin Leyton-Brown, Yatin Nandwani, Hedayat Zarkoob, Chris Cameron, Neil Newman, Dinesh Raghu, et al. Matching papers and reviewers at large conferences. arXiv preprint arXiv:2202.12273, 2022.
[21] Martin Saveski, Steven Jecmen, Nihar Shah, and Johan Ugander. Counterfactual evaluation of peer review assignment strategies in computer science and artificial intelligence. In NeurIPS, 2023.
[22] Steven Jecmen, Hanrui Zhang, Ryan Liu, Nihar Shah, Vincent Conitzer, and Fei Fang. Mitigating manipulation in peer review via randomized reviewer assignments. In NeurIPS, 2020.
[23] Eric Budish, Yeon-Koo Che, Fuhito Kojima, and Paul Milgrom. Implementing random assignments: A generalization of the Birkhoff-von Neumann theorem. In Cowles Summer Conference, 2009.
[24] OpenReview.net. OpenReview matcher. https://github.com/openreview/openreview-matcher, 2023. Accessed April 2023.
[25] Komal Dhull, Steven Jecmen, Pravesh Kothari, and Nihar B Shah. Strategyproofing peer assessment via partitioning: The price in terms of evaluators' expertise. In HCOMP, 2022.
[26] Yichong Xu, Han Zhao, Xiaofei Shi, Jeremy Zhang, and Nihar B Shah. On strategyproof conference peer review. In *IJCAI*, 2019.
[27] Steven Jecmen, Nihar B Shah, Fei Fang, and Vincent Conitzer. Tradeoffs in preventing manipulation in paper bidding for reviewer assignment. In ML Evaluation Standards Workshop at ICLR, 2022.
[28] Steven Jecmen, Minji Yoon, Vincent Conitzer, Nihar B. Shah, and Fei Fang. A dataset on malicious paper bidding in peer review. In WWW, 2023.
[29] Ruihan Wu, Chuan Guo, Felix Wu, Rahul Kidambi, Laurens Van Der Maaten, and Kilian Weinberger. Making paper reviewing robust to bid manipulation attacks. In ICML, 2021.
[30] Niclas Boehmer, Robert Bredereck, and André Nichterlein. Combating collusion rings is hard but possible. In AAAI, 2022.
[31] Andrew Tomkins, Min Zhang, and William D. Heavlin. Reviewer bias in single- versus double-blind peer review. Proceedings of the National Academy of Sciences, 2017.
[32] Neil D. Lawrence. The NIPS experiment. https://inverseprobability.com/2014/12/16/the-nips-experiment, 2014. Accessed February 1, 2023.
[33] Eric Price. The NIPS experiment. http://blog.mrtz.org/2014/12/15/the-nips-experiment.html, 2014. Accessed February 1, 2023.
[34] Ivan Stelmakh, Charvi Rastogi, Nihar B Shah, Aarti Singh, and Hal Daumé III. A large scale randomized controlled trial on herding in peer-review discussions. PLoS ONE, 2023.
[35] Alina Beygelzimer, Yann Dauphin, Percy Liang, and Jennifer Wortman Vaughan. The NeurIPS 2021 consistency experiment. https://blog.neurips.cc/2021/12/08/the-neurips-2021-consistency-experiment/, 2021. Accessed February 1, 2023.
[36] Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2023.
[37] Jack Edmonds and Richard M Karp. Theoretical improvements in algorithmic efficiency for network flow problems. Journal of the ACM (JACM), 1972.
[38] Michael B Cohen, Yin Tat Lee, and Zhao Song. Solving linear programs in the current matrix multiplication time. Journal of the ACM (JACM), 2021.
[39] Nicholas Mattei and Toby Walsh. Preflib: A library for preferences http://www.preflib.org. In ADT, 2013.
A.1 Network-Flow-Based Approximation
In Section 6, we presented Table 1, which is a shortened version of the experimental results comparing Gurobi and Algorithm 1. We now present the complete experimental results on both datasets in Tables 2 and 3 as well as the complete analysis in this section.