SlowGuess/NeurIPS_2023_OCR huggingface dataset:
G Details Regarding Assumption Suitability
In this section, we provide additional details on the discussion in Section 5 on the suitability of the monotonicity and Lipschitz smoothness assumptions.
Monotonicity. Monotonicity assumes that when any component of the covariates increases, the review quality should not be lower. We can test this assumption on the observed outcomes: among all pairs of reviewer-paper pairs with both outcomes observed,  $65.7\%$  (TPDP)/ $28.0\%$  (AAAI) have a dominance relationship  $(X_{i} \succ X_{j})$  and of those pairs,  $79.8\%$  (TPDP)/ $76.4\%$  (AAAI) satisfy the monotonicity condition. The fraction of dominant pairs for TPDP is higher since we consider only two covariates.
Lipschitz Smoothness. For the Lipschitz assumption, a choice of distance in covariate space is required. We choose the  $\ell_1$  distance, normalized in each dimension so that all component distances lie in  $[0, 1]$ , and divided by the number of dimensions. For AAAI, some reviewer-paper pairs are missing a covariate; if so, we impute a distance of 1 in that component. We then choose several potential Lipschitz constants  $L$  by analyzing the reviewer-paper pairs with observed outcomes.
First, we examine the fraction of pairs of observed reviewer-paper pairs that violate the Lipschitz condition for each value of  $L$ . Figures 3 and 4 show the CCDF of  $L$  for pairs of observations (in other words, the fraction of violating observation pairs for each value of  $L$ ) with respect to expertise and confidence respectively. In our experiments in Section 5, we use values of  $L$  corresponding to less than  $10\%$ ,  $5\%$ , and  $1\%$  violations from these plots.
Next, we examine the distances from unobserved reviewer-paper pairs to their closest observed reviewer-paper pair. In Figure 5, we show the CCDF of these distances for unobserved reviewer-paper pairs within a set of "relevant" pairs. We define the set of "relevant" unobserved pairs to be all pairs not supported on-policy that have positive probability in at least one policy among all off-policies with varying  $w_{\mathrm{text}}$  with  $q = 1$  for TPDP, and all off-policies varying  $w_{\mathrm{text}}$  and  $\lambda_{\mathrm{bid}}$  with  $q = 1$  for AAAI.
Intuitively, the Lipschitz assumptions correspond to beliefs that the outcome does not change too much as the similarity components change. As one example, for  $L = 30$  on AAAI, when one
Figure 3: CCDF of the  $L = |Y_{i} - Y_{j}| / d(X_{i},X_{j})$  values for all pairs of observed points, where  $Y_{s}$  are expertise scores. The dashed lines denote the  $L$  values corresponding to less than  $10\%$ ,  $5\%$ , and  $1\%$  violations. For TPDP, these values are  $L = 30, 50, 300$ , respectively; for AAAI,  $L = 30, 40, 100$ .
Figure 4: CCDF of the  $L = |Y_{i} - Y_{j}| / d(X_{i},X_{j})$  values for all pairs of observed points, where  $Y_{s}$  are confidence scores. The dashed lines denote the  $L$  values corresponding to less than  $10\%$ ,  $5\%$ , and  $1\%$  violations. For TPDP, these values are  $L = 30,60,400$ , respectively; for AAAI,  $L = 20,30,70$ .
Figure 5: CCDF of the distances between each relevant unobserved reviewer-paper pair and its closest observed reviewer-paper pair. The dashed lines show the medians: 0.0014 (TPDP) and 0.0011 (AAAI).
similarity component differs by 0.1, the outcomes can differ by at most 1. Effectively, the imputed outcome of each unobserved pair is restricted to be relatively close to the outcome for the closest observed pair. From the distribution in Figure 5, we observe median distances of 0.0014 (TPDP) and 0.0011 (AAAI) across the pairs violating positivity under any of these off-policies. We conclude that most imputed pairs are very close to some observed pair, and even large values of  $L$  can significantly decrease the bound width when compared to the Manski bounds.
Figure 6: The "cost of randomization" as measured by the expected total assignment similarity. The plot shows the ratio between the sum of similarities under a randomized assignment (LP (3)) with  $(q \leq 1)$  and the sum of similarities under a deterministic assignment  $(q = 1)$ . The dashed lines show the values of  $q$  set on-policy.
H Tie-Breaking Behavior
In Section 5 (Datasets), we specify a policy in terms of the parameters of LP (3) (specifically, by altering the parameters  $q$ ,  $w_{\mathrm{text}}$ , and  $\lambda_{\mathrm{bid}}$  from the on-policy values). However, LP (3) may not have a unique solution, and thus each policy may not correspond to a unique set of assignment probabilities. Of particular concern, the on-policy specification of LP (3) does not uniquely identify the actual on-policy assignment probabilities.
Ideally, we could use the same tie-breaking methodology as was used in the on-policy to pick a solution in each off-policy to avoid introducing additional effects from variations in tie-breaking behavior. However, this behavior was not specified in the venues we analyze. To resolve this, we fixed arbitrary tie-breaking behaviors such that the on-policy solution to LP (3) matches the actual on-policy assignment probabilities; we then use these same behaviors for all off-policies.
In the TPDP analyses, we perturb all similarities by small constants such that all similarity values are unique. Specifically, we change the objective of LP (3) to  $\sum_{i\in \mathcal{R}\times \mathcal{P}}P(Z_i)[(1 - \lambda)S_i + \lambda \mathcal{E}_i]$ , where  $\lambda = 1e^{-8}$ , and  $\mathcal{E}\in \mathbb{R}^{|\mathcal{R}|\times |\mathcal{P}|}$  is the same for all policies. To choose  $\mathcal{E}$ , we sampled each entry uniformly at random from  $[0,1]$  and checked that the solution of the perturbed on-policy LP matches the on-policy assignment probabilities, resampling until it does. This value of  $\mathcal{E}$  was then fixed for all policies.
In the AAAI analyses, the larger size of the similarity matrix meant that randomly choosing an  $\mathcal{E}$  that recovers the on-policy solution was not feasible. Instead, we more directly choose how to perturb the similarities in order to achieve consistency with the on-policy. We change the objective of LP (3) to  $\sum_{i\in \mathcal{R}\times \mathcal{P}}P(Z_i)(S_i - \epsilon \mathbb{I}[P_A(Z_i) = 0])$ , where  $\epsilon \in \mathbb{R}$  is chosen for each policy by the following procedure. For each policy,  $\epsilon$  is chosen to be the largest value from  $\{10^{-9},10^{-6},10^{-3}\}$  such that the difference in total similarity between the solution of the original and perturbed LPs is no greater than a tolerance of  $10^{-5}$ . We confirmed that using this procedure to perturb the on-policy LP recovers the on-policy assignment probabilities, as desired.
I Similarity Cost of Randomization
In [16], Jecmen et al. empirically analyze the "cost of randomization" in terms of the expected total assignment similarity, i.e., the objective value of LP (3), as  $q$  changes. This approach is also used by conference program chairs to choose an acceptable level of  $q$  in practice. In Figure 6, we show this trade-off between  $q$  and sum-similarity (as a ratio to the optimal deterministic sum-similarity) for both TPDP and AAAI. Note that in contrast, our approach in this work is to measure assignment quality via self-reported expertise or confidence rather than by similarity. In particular, the cost of randomization for TPDP is high in terms of sum-similarity but is revealed by our analysis to be mild in terms of expertise (Section 5, Results).
Table 1: Expertise of bad policies (95% confidence intervals).  $L = {50}$  for TPDP and  $L = {40}$  for AAAI.
Policy	Manski	Monotonicity	Lipschitz
TPDP Max	[2.6115, 2.7045]	[2.6551, 2.6782]	[2.6498, 2.6744]
TPDP Min	[2.5521, 2.6126]	[2.5521, 2.5986]	[2.5521, 2.5937]
AAAI Max	[3.3919, 3.5213]	[3.4756, 3.4783]	[3.4764, 3.4809]
AAAI Min	[3.2591, 3.3846]	[3.3394, 3.3419]	[3.3396, 3.3443]
J Power Investigation: Purposefully Bad Policies
Many of the off-policy assignments we consider in Section 5 have shown to have relatively similar estimated quality. A possible explanation for this tendency is that most "reasonable" optimized policies are roughly equivalent in terms of quality, since our analyses only consider adjusting parameters of the (presumably reasonable) optimized on-policy. To investigate this possibility, we analyze a policy intentionally chosen to have poor quality.
Designing a "bad" policy that can be feasibly analyzed presents a challenge, as the on-policies are both optimized and thus rarely place probability on obviously bad reviewer-paper pairs. To work within this constraint, we look for bad policies where all reviewer pairs with zero on-policy probability are regarded as conflicts. We then contrast the deterministic  $(q = 1)$  policy that maximizes the total similarity score with the "bad" policy that minimizes it. Since the on-policy similarities are presumably somewhat indicative of expertise, we expect the minimization policy to be worse.
The results of this comparison are presented in Table 1. On both TPDP and AAAI, we see that our methods clearly identify the minimization policies as worse. The differences in quality between the policies becomes clearer with the addition of Lipschitz and monotonicity assumptions to address attrition. This illustrates that our methods are able to distinguish a good policy (the best of the best matches) from a clearly worse one (the worst of the best matches). Thus, it is likely that our primary analyses are simply exploring high-quality regions of the assignment-policy space, and that peer review assignment quality is often robust to the exact values of the various parameters.
K Broader Impact
We hope that our work will have a substantial positive impact on the quality of peer review processes broadly. Our methodology provides conference program chairs and other organizers with a way to evaluate the quality of alternative reviewer-paper assignments that they can easily use for postmortem analysis. The results from such analyses can be leveraged to adjust the assignment algorithm for future iterations of the conference, ideally leading to improved review quality and improved satisfaction from both reviewers and authors.
We note two potential negative impacts that our work may have. First, our methodology requires making assumptions about the unobserved outcomes (e.g., monotonicity, Lipschitz smoothness) in order to achieve good estimates of assignment quality. If these assumptions do not hold in practice, the resulting estimates may be misleading, potentially supporting adjustments to the assignment policy that in fact decrease review quality. However, as there currently is no principled method for evaluating the quality of alternative assignment policies, the status quo is that such assignment policy choices are made based primarily on the intuition of program chairs. Our work makes the underlying assumptions explicit so that they can be directly considered.
Second, our work considers estimates of the average review quality across all assigned reviewer-paper pairs, analogous to the common sum-of-similarities objective used for review-paper assignment (see Section 2). However, past work has also considered other assignment objectives based on the fairness of the assignment [35, 36, 47, 48]. By focusing only on the average review quality, our analysis does not consider the impact on individual reviewers or papers, potentially encouraging assignment policies that produce higher overall review quality at the cost of fairness. In this work, we focused on average review quality as it is the primary metric currently used by conference organizers, but consider generalizing our approach to estimands based on other notions of fairness a very important direction for future work.
L Results for Confidence Outcomes
Figure 7 shows the results of our analyses using confidence as a quality measure  $(Y)$ . We find that the results are substantively very similar to those reported in Section 5 using expertise.
Figure 7: Confidence of off-policies varying  $w_{\mathrm{text}}$  and  $q$  for TPDP and  $w_{\mathrm{text}}$ ,  $\lambda_{\mathrm{bid}}$ , and  $q$  for AAAI, computed using the different estimation methods described in Section 4. The dashed blue lines indicate Manski bounds around the on-policy confidence, and the grey lines indicate Manski bounds around the off-policy confidence. The error bands (denoted  $\widehat{\mu}_B^{CI}$  for the Manski bounds,  $\widehat{\mu}_{B|M}^{CI}$  for the monotonicity bounds, and  $\widehat{\mu}_{B|L}^{CI}$  for the Lipschitz bounds) represent confidence intervals that asymptotically contain the true value of  $\mu_B$  with probability at least 95% as described in Appendix C. Note that to focus on the most relevant regions of the plots, the vertical axes do not start at zero.