SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Future Work. In the context of peer review, the present work considers only a few parameterized slices of the vast space of reviewer-paper assignment policies, while there are many other substantive questions that our methodology can be used to answer. For instance, one could evaluate assignment quality under a different method of computing similarity scores (e.g., different NLP algorithms [46]), additional constraints on the assignment (e.g., based on seniority or geographic diversity [6]), or objective functions other than the sum-of-similarities (e.g., various fairness-based objectives [35, 36, 47, 48]). Additional thought should also be given to the trade-offs between maximizing review quality and broader considerations of reviewer welfare: while assignments based on high text similarity may yield slightly higher-quality reviews, reviewers may be more willing to review again if the assignment policy follows their bids more closely. Beyond peer review, our work is applicable to off-policy evaluation in other matching problems, including education [49, 50], advertising [21], and ride-sharing [22].
We thank Gautam Kamath and Rachel Cummings for allowing us to conduct this study in TPDP and Melisa Bok and Celeste Martinez Gomez from OpenReview for helping us with the OpenReview APIs. We are also grateful to Samir Khan and Tal Wagner for helpful discussions. This work was supported in part by NSF CAREER Award 2143176, NSF CAREER Award 1942124, NSF CIF 1763734, and ONR N000142212181.
[1] Jim McCullough. First comprehensive survey of NSF applicants focuses on their concerns about proposal review. Science, Technology, & Human Values, 1989.
[2] Marko A. Rodriguez, Johan Bollen, and Herbert Van de Sompel. Mapping the bid behavior of conference referees. Journal of Informetrics, 1(1):68-82, 2007.
[3] Terne Thorn Jakobsen and Anna Rogers. What factors should paper-reviewer assignments rely on? community perspectives on issues and ideals in conference peer-review. In Conference of the North American Chapter of the Association for Computational Linguistics, pages 4810-4823, 2022.
[4] Nihar B. Shah. Challenges, experiments, and computational solutions in peer review. Communications of the ACM, 65(6):76-87, 2022.
[5] Laurent Charlin, Richard S. Zemel, and Craig Boutilier. A framework for optimizing paper matching. In Uncertainty in Artificial Intelligence, volume 11, pages 86-95, 2011.
[6] Kevin Leyton-Brown, Mausam, Yatin Nandwani, Hedayat Zarkoob, Chris Cameron, Neil Newman, and Dinesh Raghu. Matching papers and reviewers at large conferences. arXiv preprint arXiv:2202.12273, 2022.
[7] Simon Price and Peter A. Flach. Computational support for academic peer review: A perspective from artificial intelligence. Communications of the ACM, 60(3):70-79, 2017.
[8] Baochun Li and Y Thomas Hou. The new automated IEEE INFOCOM review assignment system. IEEE Network, 30(5):18-24, 2016.
[9] Neil D. Lawrence. The NIPS experiment. https://inverseprobability.com/2014/12/16/the-nips-experiment, 2014. Accessed May 17, 2023.
[10] Eric Price. The NIPS experiment. http://blog.mrtz.org/2014/12/15/the-nips-experiment.html, 2014. Accessed May 17, 2023.
[11] Andrew Tomkins, Min Zhang, and William D. Heavlin. Reviewer bias in single- versus double-blind peer review. Proceedings of the National Academy of Sciences, 114(48):12708-12713, 2017.
[12] Ivan Stelmakh, Charvi Rastogi, Nihar B Shah, Aarti Singh, and Hal Daume III. A large scale randomized controlled trial on herding in peer-review discussions. Plos one, 18(7):e0287443, 2023.
[13] Alina Beygelzimer, Yann Dauphin, Percy Liang, and Jennifer Wortman Vaughan. The NeurIPS 2021 consistency experiment. https://blog.neurips.cc/2021/12/08/the-neurips-2021-consistency-experiment/, 2021. Accessed May 17, 2023.
[14] Mathias Lecuyer, Joshua Lockerman, Lamont Nelson, Siddhartha Sen, Amit Sharma, and Aleksandrs Slivkins. Harvesting randomness to optimize distributed systems. In ACM Workshop on Hot Topics in Networks, pages 178-184, 2017.
[15] Michael Littman. Collusion rings threaten the integrity of computer science research. Communications of the ACM, 2021.
[16] Steven Jecmen, Hanrui Zhang, Ryan Liu, Nihar B. Shah, Vincent Conitzer, and Fei Fang. Mitigating manipulation in peer review via randomized reviewer assignments. Advances in Neural Information Processing Systems, 2020.
[17] Charles F Manski. Nonparametric bounds on treatment effects. The American Economic Review, 80(2):319-323, 1990.
[18] Noveen Sachdeva, Yi Su, and Thorsten Joachims. Off-policy bandits with deficient support. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 965–975, 2020.
Counterfactual Evaluation of Peer-Review Assignment Policies Supplemental Material
Martin Saveski, Steven Jecmen, Nihar B. Shah, Johan Ugander
A Linear Programs for Peer-Review Assignment
Deterministic Assignment. Let  $Z \in \{0,1\}^{|\mathcal{R}| \times |\mathcal{P}|}$  be an assignment matrix where  $Z_{r,p}$  denotes whether reviewer  $r \in \mathcal{R}$  is assigned to paper  $p \in \mathcal{P}$ . Given a matrix of similarity scores  $S \in \mathbb{R}_{\geq 0}^{|\mathcal{R}| \times |\mathcal{P}|}$ , a standard objective is to find an assignment of papers to reviewers that maximizes the sum of similarities of the assigned pairs, subject to constraints that each paper is assigned to an appropriate number of reviewers  $\ell$ , each reviewer is assigned no more than a maximum number of papers  $k$ , and conflicts of interest are respected [5, 27, 30-34]. Denoting the set of conflict-of-interest pairs by  $\mathcal{C} \subset \mathcal{R} \times \mathcal{P}$ , this optimization problem can be formulated as the following linear program:
$$
\max _ {Z _ {r, p}: r \in \mathcal {R}, p \in \mathcal {P}} \sum_ {r \in \mathcal {R}, p \in \mathcal {P}} Z _ {r, p} S _ {r, p}
$$
s.t.  $\sum_{r\in \mathcal{R}}Z_{r,p} = \ell$ $\forall p\in \mathcal{P}$
$$
\sum_ {p \in \mathcal {P}} Z _ {r, p} \leq k \quad \forall r \in \mathcal {R}
$$
$$
Z _ {r, p} = 0 \quad \forall (r, p) \in \mathcal {C}
$$
$$
0 \leq Z _ {r, p} \leq 1 \quad \forall r \in \mathcal {R}, p \in \mathcal {P}.
$$
By total unimodularity conditions, this problem has an optimal solution where  $Z_{r,p} \in \{0,1\}$ ,  $\forall r \in \mathcal{R}, p \in \mathcal{P}$ .
Although the above strategy is the primary method used for paper assignments in large-scale peer review, other variants of this method have been proposed and used in the literature. These algorithms consider various properties in addition to the total similarity, such as fairness [35, 36], strategyprofness [37, 51], envy-freeness [47] and diversity [52]. We focus on the sum-of-similarities objective here, but our off-policy evaluation framework is agnostic to the specific objective function.
Randomized Assignment. As one approach to strategyproofness, Jecmen et al. [16] introduce the idea of using randomization to prevent colluding reviewers and authors from being able to guarantee their assignments. Specifically, the algorithm computes a randomized paper assignment, where the marginal probability  $P(Z_{r,p})$  of assigning any reviewer  $r$  to any paper  $p$  is at most a parameter  $q \in [0,1]$ , chosen a priori by the program chairs. These marginal probabilities are determined by the following linear program, which maximizes the expected similarity of the assignment:
$$
\max  _ {P \left(Z _ {r, p}\right): r \in \mathcal {R}, p \in \mathcal {P}} \sum_ {r \in \mathcal {R}, p \in \mathcal {P}} P \left(Z _ {r, p}\right) S _ {r, p} \tag {3}
$$
s.t.  $\sum_{r\in \mathcal{R}}P(Z_{r,p}) = \ell$ $\forall p\in \mathcal{P}$
$$
\sum_ {p \in \mathcal {P}} P (Z _ {r, p}) \leq k \quad \forall r \in \mathcal {R}
$$
$$
P (Z _ {r, p}) = 0 \quad \forall (r, p) \in \mathcal {C}
$$
$$
0 \leq P (Z _ {r, p}) \leq q \quad \forall r \in \mathcal {R}, p \in \mathcal {P}.
$$
A reviewer-paper assignment is then sampled using a randomized procedure that iteratively redistributes the probability mass placed on each reviewer-paper pair until all probabilities are either zero or one. This procedure ensures only that the desired marginal assignment probabilities are satisfied, providing no guarantees on the joint distributions of assigned pairs.
B “No Interference” Assumption
Our estimators assume that there is no interference between the units, i.e., that the treatment of one unit does not affect the outcomes for the other units. In the causal inference literature, this assumption
is referred to as the Stable Unit Treatment Value Assumption (SUTVA) [53]. In the context of peer review, SUTVA implies that: (i) The quality  $Y_{r,p}$  of the review by reviewer  $r$  reviewing paper  $p$  does not depend on what other reviewers are assigned to paper  $p$ ; and (ii) the quality also does not depend on the other papers that reviewer  $r$  was assigned to review. The first assumption is quite realistic as in most peer review systems the reviewers cannot see other reviews until they submit their own. The second assumption is important to understand, as there could be "batch effects": a reviewer may feel more or less confident about their assessment (if measuring quality by confidence) depending on what other papers they were assigned to review. We do not test for batch effects or other violations of SUTVA in this work, which typically require either strong modeling assumptions or complex experimental designs [54-56] specifically tailored for testing SUTVA, but consider it important future work.
C Variance Computation and Confidence Interval Construction
In this section, we discuss how we construct confidence intervals around  $\mu_B$  under Manski, monotonicity, and Lipschitz assumptions. To construct confidence intervals, we estimate the variance of  $\widehat{\mu}_B(Y^{l_mpute})$  as follows:
$$
\widehat {\operatorname {V a r}} \left[ \widehat {\mu} _ {B} \left(Y ^ {\text {I m p u t e}}\right) \right] = \frac {1}{N ^ {2}} \sum_ {(i, j) \in (\mathcal {R} \times \mathcal {P}) ^ {2}} \operatorname {C o v} \left[ Z _ {i}, Z _ {j} \right] Z _ {i} ^ {A} Z _ {j} ^ {A} W _ {i} W _ {j} Y _ {i} ^ {\prime} Y _ {j} ^ {\prime},
$$
$$
\text {w h e r e} Y _ {i} ^ {\prime} = \left\{ \begin{array}{l l} Y _ {i} & \text {i f} i \in \mathcal {I} ^ {+} \\ Y _ {i} ^ {\text {I m p u t e}} & \text {i f} i \in \mathcal {I} ^ {A t t} \cup \mathcal {I} ^ {-} \\ \overline {{Y}} & \text {i f} i \in \mathcal {I} ^ {A b s}. \end{array} \right.
$$
The covariance terms (taken over  $Z \sim P_A$ ) are not known exactly, owing to the fact that the procedure by Jecmen et al. [16] only constrains the marginal probabilities of individual reviewer-paper pairs, but pairs of pairs can be non-trivially correlated. In the absence of a closed-form expression, we use Monte Carlo methods to tightly estimate these covariances. In both our analyses of the TPDP and AAAI datasets, we sampled 1 million assignments and computed the empirical covariance. We ran an additional analysis to investigate the variability of our variance estimates. We took a bootstrap sample of 100,000 assignments (from the set of all 1 million assignments we sampled) and computed the variance based only on the (smaller) bootstrap sample. We repeated this procedure 1,000 times and computed the variance of our variance estimates. We found that the variance of our variance estimates is very small (less than  $10^{-9}$ ) even when we use 10 times fewer sampled assignments, suggesting that we have sampled enough assignments to accurately estimate the variance.
When employing Manski bounds, we can adapt a well-established inference procedure to construct  $95\%$  confidence intervals of a partial identification region that asymptotically contain the true value of  $\mu_B$  with probability at least  $95\%$ . Following Imbens and Manski [43], we construct the interval:
$$
\widehat {\mu} _ {B} ^ {C I} \in \left[ \widehat {\mu} _ {B} (y _ {\mathrm {m i n}}) - z _ {\alpha , n} ^ {\prime} \sqrt {\widehat {\mathbf {V a r}} [ \widehat {\mu} _ {B} (y _ {\mathrm {m i n}}) ] / N}, \widehat {\mu} _ {B} (y _ {\mathrm {m a x}}) + z _ {\alpha , n} ^ {\prime} \sqrt {\widehat {\mathbf {V a r}} [ \widehat {\mu} _ {B} (y _ {\mathrm {m a x}}) ] / N} \right],
$$
where the  $z$ -score analog  $z_{\alpha, n}^{\prime} (\alpha = 0.95)$ , is set by their procedure such that the interval asymptotically has at least  $95\%$  coverage under plausible regularity conditions; for further details, see the discussion below. When employing the monotonicity assumption, we construct the interval:
$$
\widehat {\mu} _ {B | M} ^ {C I} \in \left[ \widehat {\mu} _ {B} (\widetilde {Y} _ {m i n} ^ {M o n}) - z _ {\alpha , n} ^ {\prime} \sqrt {\widehat {\mathrm {V a r}} [ \widehat {\mu} _ {B} (\widetilde {Y} _ {m i n} ^ {M o n}) ] / N}, \widehat {\mu} _ {B} (\widetilde {Y} _ {m a x} ^ {M o n}) + z _ {\alpha , n} ^ {\prime} \sqrt {\widehat {\mathrm {V a r}} [ \widehat {\mu} _ {B} (\widetilde {Y} _ {m a x} ^ {M o n}) ] / N} \right],
$$
and when employing the Lipschitz assumption, we construct the interval:
$$
\widehat {\mu} _ {B | L} ^ {C I} \in \left[ \widehat {\mu} _ {B} (\widetilde {Y} _ {m i n} ^ {L}) - z _ {\alpha , n} ^ {\prime} \sqrt {\widehat {\mathrm {V a r}} [ \widehat {\mu} _ {B} (\widetilde {Y} _ {m i n} ^ {L}) ] / N}, \widehat {\mu} _ {B} (\widetilde {Y} _ {m a x} ^ {L}) + z _ {\alpha , n} ^ {\prime} \sqrt {\widehat {\mathrm {V a r}} [ \widehat {\mu} _ {B} (\widetilde {Y} _ {m a x} ^ {L}) ] / N} \right].
$$
These intervals converge uniformly to the specified  $\alpha$ -level coverage under a set of regularity assumptions on the behavior of the estimators of the upper and lower endpoints of the interval estimate: Assumption 1 from [43], establishing the coverage result in Lemma 4 there. It is difficult to verify whether Assumption 1 is satisfied for the designs (sampling reviewer-paper matchings) and interval endpoint estimators (Manski, monotonocity, Lipschitz) in this work.
A different set of assumptions, most significantly that the fraction of missing data is known before assignment, support a different method for computing confidence intervals with the coverage result in Lemma 3 from [43], obviating the need for Assumption 1. In our setting, small amounts of attrition (relative to the number of policy-induced positivity violations) mean that the fraction of data that is missing is not exactly known before assignment, but almost. In practice, we find that the Imbens-Manski interval estimates from their Lemma 3 (assuming a known fraction of missing data) and Lemma 4 (assuming Assumption 1) are nearly identical for all three of the Manksi-, monotonicity-, and Lipschitz-based estimates, suggesting the coverage is well-behaved. A detailed theoretical analysis of whether the estimators obey the regularity conditions of Assumption 1 is beyond the scope of this work.
D Linear Programs for Partial Identification Based on Lipschitz Smoothness
In this section, we provide a more detailed description of the linear programs used to compute our partial identification estimates assuming Lipschitz smoothness on the correspondence between covariates and outcomes. Intuitively, the Lipschitz smoothness assumption captures the idea that we expect two reviewer-paper pairs that are very similar in covariate space to have similar expertise. For covariate vectors  $X_{i}$  and  $X_{j}$ , define  $d(X_{i},X_{j})$  as some notion of distance between the covariates. Then, the Lipschitz assumption states that there exists a constant  $L$  such that  $|Y_{i} - Y_{j}| \leq Ld(X_{i},X_{j})$  for all  $i$  and  $j$  in  $\mathcal{R} \times \mathcal{P}$ .
As in our formulation for partial identification based on monotonicity (Section 4, LP (2)), we introduce surrogate values  $\widetilde{Y}_i$  and implement a two-level optimization problem to address Lipschitz violations within the observed outcomes, i.e., if two observed pairs are very close in covariate space but have different outcomes. We also define the universe of relevant pairs  $\mathcal{U} = \mathcal{O} \cup \mathcal{I}^{Att} \cup \mathcal{I}^{Abs} \cup \mathcal{I}^{-}$  and a very large constant  $\Psi$ .
This results in the following pair of optimization problems:
$$
\begin{array}{l} \left(\widetilde {Y} _ {m i n} ^ {L}, \widetilde {Y} _ {m a x} ^ {L}\right) = \underset {\widetilde {Y} _ {i}: i \in \mathcal {U}} {\operatorname {a r g m i n}} \Psi \sum_ {i \in \mathcal {O}} | \widetilde {Y} _ {i} - Y _ {i} | \pm \left(\sum_ {i \in \mathcal {I} ^ {A t t} \cup \mathcal {I} ^ {A b s}} \widetilde {Y} _ {i} W _ {i} + \sum_ {i \in \mathcal {I} ^ {-}} \widetilde {Y} _ {i} P _ {B} (Z _ {i})\right) \\ \begin{array}{l l} \text {s . t .} & | \widetilde {Y} _ {i} - \widetilde {Y} _ {j} | \leq L d (X _ {i}, X _ {j}) \quad \forall (i, j) \in \mathcal {U}, \end{array} \\ y _ {m i n} \leq \widetilde {Y} _ {i} \leq y _ {m a x} \quad \forall i \in \mathcal {U}. \\ \end{array}
$$
The sign of the second objective term depends on whether a lower (negative) or upper (positive) bound is being computed. The last set of constraints are the same constraints used to construct the Manski bounds described in Section 4, which here are combined with the Lipschitz assumption to jointly constrain the possible outcomes. In the limit, as  $L \to \infty$ , the Lipschitz constraints become vacuous and we recover the Manski bounds. This problem can be reformulated and solved as a linear program using standard techniques.
E Model Implementation
To impute the outcomes of the unobserved reviewer-paper pairs, we train classification, ordinal regression, and collaborative filtering models. Classification models are suitable since the reviewers select their expertise and confidence scores from a set of pre-specified choices. Ordinal regression models additionally model the fact that the scores have a natural ordering. Collaborative filtering models, in contrast to the classification and ordinal regression models, do not rely on covariates and instead model the structure of the observed entries in the reviewer-paper outcome matrix, which is akin to user-item rating matrices found in recommender systems.
In the classification and regression models, we use the covariates  $X_{i}$  for each reviewer-paper pair as input features. In our analysis, we consider the two/three component scores used to compute the similarities: for TPDP,  $X_{i} = (T_{i},B_{i})$ ; for AAAI,  $X_{i} = (T_{i},K_{i},B_{i})$ . These are the primary components used by conference organizers to compute similarities, so we expect them to be usefully correlated with match quality. Although we perform our analysis with this choice of covariates, one
Test Mean Absolute Error
Figure 2: Test performance of the imputation models described in Section 4 (Model Imputation), averaged across 10 random train/test splits of all observed reviewer-paper pairs. The error bars show  $95\%$  confidence intervals.
could also include various other features of each reviewer-paper pair, e.g., some encoding of reviewer and paper subject areas, reviewer seniority, etc.
To evaluate the performance of the models, we randomly split the observed reviewer-paper pairs into train (75%) and test (25%) sets, fit the models on the train set, and measure the mean absolute error (MAE) of the predictions on the test set. To get more robust estimates of the performance, we repeat this process 10 times. In the training phase, we use 10-fold cross-validation to tune the hyperparameters, using MAE as a selection criterion, and retrain the model on the full training set with the best hyperparameters. We also consider two preprocessing decisions: (a) whether to encode the bids as one-hot categorical variables or continuous variables with the values described in Section 5, and (b) whether to standardize the features. In both cases, we used the settings that, overall, worked best (at prediction) for each model. We tested several models from each model category. To simplify the exposition, we only report the results of the two best-performing models in each category. The code repository referenced in Section 1 contains the implementation of all models, including the sets of hyperparameters considered for each model.
Figure 2 shows the test MAE across the 10 random train/test splits (means and  $95\%$  CIs) using expertise and confidence outcomes for both TPDP and AAAI. We note that all models perform significantly better than a baseline that predicts the mean outcome in the train set. For TPDP, we find that all models perform similarly, except for  $cf-svd++$ , which performs slightly better than the other models, both for expertise and confidence. For AAAI, all classification and regression models perform similarly, but the collaborative filtering models perform slightly worse. This difference in performance is perhaps due to the fact that we consider a larger set of covariates for AAAI than TPDP, likely making the classification and ordinal regression models more predictive.
Finally, to estimate  $\widehat{\mu}_B$ , we train each model on the set of all observed reviewer-paper pairs, predict the outcomes for all unobserved pairs, and impute the predicted outcomes as described in Section 4. In the training phase, we use 10-fold cross-validation to select the hyperparameters and refit the model on the full set of observed reviewer-paper pairs.
F Details of AAAI Assignment
In Section 5, we described a simplified version of the stage of the AAAI assignment procedure that we analyze, i.e., the assignment of senior reviewers to the first round of submissions. In this section, we describe this stage of the AAAI paper assignment more precisely.
A randomized assignment was computed between 3145 senior reviewers and 8450 first-round paper submissions, independent of all other stages of the reviewer assignment. The set of senior reviewers was determined based on reviewing experience and publication record; these criteria were external to the assignment. Each paper was assigned  $\ell = 1$  senior reviewer. Reviewers were assigned to at
most  $k = 4$  papers, with the exception of reviewers with a "Machine Learning" primary area or in the "AI For Social Impact" track, who were assigned to at most  $k = 3$  papers. The probability limit was  $q = 0.52$ .
The similarities were computed from text-similarity scores  $T_{i}$ , subject-area scores  $K_{i}$ , and bids  $B_{i}$ . Either the text-similarity scores or the area scores could be missing for a given reviewer-paper pair, due to either a reviewer failing to provide the needed information or due to other errors in the computation of the scores. The text-similarity scores  $T_{i}$  were created using text-based scores from two different sources: (i) the Toronto Paper Matching System (TPMS) [27], and (ii) the ACL Reviewer Matching code [28]. The text-similarity scores  $T_{i}$  was set equal to the TPMS score for all pairs where this score was not missing, set equal to the ACL score for all other pairs where the ACL score was not missing, and marked as missing if both scores were missing. The subject-area scores were computed from reviewer and paper subject areas using the procedure described in Appendix A of [6].
Next, base scores  $S_{i}^{\prime} = w_{\mathrm{text}}T_{i} + (1 - w_{\mathrm{text}})K_{i}$  were then computed with  $w_{\mathrm{text}} = 0.75$ , if both  $T_{i}$  and  $K_{i}$  were not missing. If either  $T_{i}$  or  $K_{i}$  was missing, the base score was equal to the non-missing score of the two. If both were missing, the base score was set as  $S_{i}^{\prime} = 0$ . For pairs where the bid was "willing" or "eager" and  $K_{i} = 0$ , the base score was set as  $S_{i}^{\prime} = T_{i}$ .
Next, final scores were computed as  $S_{i} = S_{i}^{\prime 1 / B_{i}}$ , using the bid values "not willing" (0.05), "not entered" (1), "in a pinch" (1 + 0.5λbids), "willing" (1 + 1.5λbids), "eager" (1 + 3λbids); with λbids = 1. If  $S_{i} < 0.15$  and  $K_{i}$  was not missing, the final score was recomputed as  $S_{i} = \min(K_{i}^{1 / B_{i}}, 0.15)$ . Finally, for reviewers who did not provide their profile for use in conflict-of-interest detection, the final score was reduced by 10%.
In all of our analyses, we follow this same procedure to determine the assignment under alternative policies (varying only the parameters  $w_{\mathrm{text}}$ ,  $\lambda_{\mathrm{bid}}$ , and  $q$ ).