SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mean Imputation. As a first approach, we assume that the mean outcome within  $\mathcal{I}^+$  is representative of the mean outcome among the other pairs. This is a strong assumption, since the presence of a pair in  $\mathcal{I}^-$  or  $\mathcal{I}^{Att}$  may not be independent of their outcome. For example, if reviewers choose not to submit reviews when the assignment quality is poor,  $\overline{Y}$  is not representative of the outcomes in  $\mathcal{I}^{Att}$ . Nonetheless, under this strong assumption, we can simply impute the mean outcome  $\overline{Y}$  for all pairs necessitating imputation. Setting  $Y_{i}^{Impute} = \overline{Y}$  for all  $i\in \mathcal{I}^{-}\cup \mathcal{I}^{Att}$ , we consider the following point estimate of  $\mu_B$ :  $\widehat{\mu}_B(\overline{Y})$ . While following from an overly strong assumption, we find it useful to compare our findings under this assumption to findings under subsequent weaker assumptions.
Model Imputation. Instead of simply imputing the mean outcome, we can assume that the unobserved outcomes  $Y_{i}$  are some simple function of known covariates  $X_{i}$  for each reviewer-paper
pair  $i$ . If so, we can directly estimate this function using a variety of statistical models, resulting in a point estimate of  $\mu_B$ . In doing so, we implicitly take on the assumptions made by each model, which determine how to generalize the covariate-outcome mapping from the observed pairs to the unobserved pairs. These assumptions are typically quite strong, since this mapping may be very different between the observed pairs (typically good matches) and unobserved pairs (typically less good matches).
Specifically, given the set of all observed reviewer-paper pairs  $\mathcal{O} = \{i\in \mathcal{I}^{+}:Z_{i}^{A} = 1\}$ , we train a model  $m$  using the observed data  $\{(X_i,Y_i):i\in \mathcal{O}\}$ . Let  $\widehat{Y}^{(m)}\in \mathbb{R}^{|\mathcal{R}|\times |\mathcal{P}|}$  denote the outcomes predicted by that model for each pair. We then consider  $\widehat{\mu}_B(\widehat{Y}^{(m)})$  as a point estimate of  $\mu_B$ . In our experiments, we employ standard methods for classification, ordinal regression, and collaborative filtering: logistic regression (clf-logistic); ridge classification (clf-ridge); ordered logit (ord-logit); ordered probit (ord-probit); SVD++ collaborative filtering (cf-svd++.); and K-nearest-neighbors collaborative filtering (cf-knn). Note that, unlike the other methods, the methods based on collaborative filtering model the missing data by using only the observed reviewer-paper outcomes  $(Y)$ . We discuss our choice of methods, hyperparameters, and implementation details in Appendix E.
Manski Bounds. As a more conservative approach, we can exploit the fact that the outcomes  $Y_{i}$  are bounded, letting us bound the mean of the counterfactual policy without making any assumptions on how the positivity violations arise. Such bounds are often called Manski bounds [17] in the econometrics literature on partial identification. To employ Manski bounds, we assume that all outcomes  $Y$  can take only values between  $y_{\mathrm{min}}$  and  $y_{\mathrm{max}}$ , e.g., self-reported expertise and confidence scores are limited to a pre-specified range on the review questionnaire. Then, setting  $Y_{i}^{Impute} = y_{\mathrm{min}}$  or  $Y_{i}^{Impute} = y_{\mathrm{max}}$  for all  $i \in \mathcal{I}^{-} \cup \mathcal{I}^{Att}$ , we can estimate the upper and lower bound of  $\mu_B$  as  $\widehat{\mu}_B(y_{\mathrm{min}})$  and  $\widehat{\mu}_B(y_{\mathrm{max}})$ . In Appendix C, we discuss how we construct confidence intervals for these estimates that asymptotically contain the true value of  $\mu_B$  with probability at least 95% by adapting the inference procedure proposed in [43].
Monotonicity and Lipschitz Smoothness. We now propose two styles of weak assumptions on the covariate-outcome mapping that can be leveraged to achieve tighter bounds on  $\hat{\mu}_B$  than the Manski bounds. In contrast to the strong modeling assumptions used in the mean and model imputation, these assumptions can be more intuitively understood and justified as conservative assumptions given particular choices of covariates.
Monotonicity. The first weak assumption we consider is a monotonicity condition. Intuitively, monotonicity captures the idea that we expect higher expertise for a reviewer-paper pair when some covariates are higher, all else equal. For example, in our experiments, we use the similarity component scores (bids, text similarity, subject area match) as covariates. Specifically, for covariate vectors  $X_{i}$  and  $X_{j}$ , define the dominance relationship  $X_{i} \succ X_{j}$  to mean that  $X_{i}$  is greater than or equal to  $X_{j}$  in all components and  $X_{i}$  is strictly greater than  $X_{j}$  in at least one component. Then, the monotonicity assumption states that: if  $X_{i} \succ X_{j}$ , then  $Y_{i} \geq Y_{j}, \forall (i,j) \in (\mathcal{R} \times \mathcal{P})^{2}$ .
Using this assumption to restrict the range of possible values for the unobserved outcomes, we seek upper and lower bounds on  $\mu_B$ . Recall that  $\mathcal{O}$  is the set of all observed reviewer-paper pairs. One challenge is that the observed outcomes themselves ( $Y_i$  for  $i \in \mathcal{O}$ ) may violate the monotonicity condition. Thus, to find an upper or lower bound, we compute surrogate values  $\widetilde{Y}_i \in \mathbb{R}$  that satisfy the monotonicity constraint for all  $i \in \mathcal{O} \cup \mathcal{I}^{Att} \cup \mathcal{I}^{Abs} \cup \mathcal{I}^{-}$  while ensuring that the surrogate values  $\widetilde{Y}_i$  for  $i \in \mathcal{O}$  are as close as possible to the outcomes  $Y_i$ . The surrogate values  $\widetilde{Y}_i$  for  $i \in \mathcal{I}^{Att} \cup \mathcal{I}^{-}$  can then be imputed as outcomes.
Inspired by isotonic regression [44], we implement a two-level optimization problem. The primary objective minimizes the  $\ell_1$  distance between  $\widetilde{Y}_i$  and  $Y_{i}$  for pairs with observed outcomes  $i\in \mathcal{O}$ . The second objective either minimizes (for a lower bound) or maximizes (for an upper bound) the sum of  $Y_{i}$  for the unobserved pairs  $i\in \mathcal{I}^{Att}\cup \mathcal{I}^{Abs}\cup \mathcal{I}^{-}$ , weighted as in  $\widehat{\mu}_B$ . Define the universe of relevant pairs  $\mathcal{U} = \mathcal{O}\cup \mathcal{I}^{Att}\cup \mathcal{I}^{Abs}\cup \mathcal{I}^{-}$  and define  $\Psi$  as a very large constant. This results in the following
pair of optimization problems:
$$
\begin{array}{l} \left(\widetilde {Y} _ {\min } ^ {M}, \widetilde {Y} _ {\max } ^ {M}\right) = \underset {\widetilde {Y} _ {i}: i \in \mathcal {U}} {\operatorname {a r g m i n}} \quad \Psi \sum_ {i \in \mathcal {O}} \left| \widetilde {Y} _ {i} - Y _ {i} \right| \pm \left(\sum_ {i \in \mathcal {I} ^ {A t t} \cup \mathcal {I} ^ {A b s}} \widetilde {Y} _ {i} W _ {i} + \sum_ {i \in \mathcal {I} ^ {-}} \widetilde {Y} _ {i} P _ {B} (Z _ {i})\right), \tag {2} \\ \begin{array}{r l} \mathrm {s . t .} & \widetilde {Y} _ {i} \geq \widetilde {Y} _ {j} \quad \forall (i, j) \in \{\mathcal {U} ^ {2}: X _ {i} \succ X _ {j} \}, \qquad y _ {\min } \leq \widetilde {Y} _ {i} \leq y _ {\max } \quad \forall i \in \mathcal {U}. \end{array} \\ \end{array}
$$
The sign of the second objective term depends on whether a lower (negative) or upper (positive) bound is being computed. The last set of constraints corresponds to the same constraints used to construct the Manski bounds described earlier, which is combined here with monotonicity to jointly constrain the possible outcomes. The above problem can be reformulated and solved as a linear program using standard techniques. We can then estimate the upper and lower bound of  $\mu_B$  as  $\widehat{\mu}_B(\widetilde{Y}_{min}^{Mon})$  and  $\widehat{\mu}_B(\widetilde{Y}_{max}^{Mon})$ , and construct  $95\%$  confidence intervals as discussed in Appendix C.
Lipschitz Smoothness. The second weak assumption we consider is a Lipschitz smoothness assumption on the correspondence between covariates and outcomes. Intuitively, this captures the idea that we expect two reviewer-paper pairs who are very similar in covariate space to have similar expertise. For covariate vectors  $X_{i}$  and  $X_{j}$ , define  $d(X_{i},X_{j})$  as some notion of distance between the covariates. Then, the Lipschitz assumption states that there exists a constant  $L$  such that  $|Y_{i} - Y_{j}| \leq Ld(X_{i},X_{j})$  for all  $(i,j) \in (\mathcal{R} \times \mathcal{P})^{2}$ . In practice, we can choose an appropriate value of  $L$  by studying the many pairs of observed outcomes in the data (Section 5 and Appendix G), though this approach assumes that the Lipschitz smoothness of the covariate-outcome function is the same for observed and unobserved pairs.
As in the previous section, we introduce surrogate values  $\widetilde{Y}_i\in \mathbb{R}$  and implement a two-level optimization problem to address Lipschitz violations within the observed outcomes (i.e., if two observed pairs are very close in covariate space but have different outcomes). The resulting pair of optimization problems is defined exactly as in problem (2), except that the first set of constraints is replaced with constraints  $|\widetilde{Y}_i - \widetilde{Y}_j|\leq Ld(X_i,X_j),\forall (i,j)\in \mathcal{U}$ . We include the complete LP formulation in Appendix D. We denote the solutions to these problems by  $\widetilde{Y}_{min}^{L}$  and  $\widetilde{Y}_{max}^{L}$ . As before, the sign of the second objective term depends on whether a lower (negative) or upper (positive) bound is being computed. The last set of constraints are again the same constraints used to construct the Manski bounds described earlier, which here are combined with the Lipschitz assumption to jointly constrain the possible outcomes. In the limit, as  $L\to \infty$ , the Lipschitz constraints become vacuous and we recover the Manski bounds. This problem can again be reformulated and solved as a linear program using standard techniques. We can then estimate the upper and lower bound of  $\mu_B$  as  $\widehat{\mu}_B(\widetilde{Y}_{min}^L)$  and  $\widehat{\mu}_B(\widetilde{Y}_{max}^L)$ , and construct  $95\%$  confidence intervals as discussed in Appendix C.
We apply our framework to data from two venues that used randomized paper assignments as described in Section 2, the 2021 Workshop on Theory and Practice of Differential Privacy (TPDP) and the 2022 AAAI Conference on Advancement in Artificial Intelligence (AAAI). In both settings, we aim to understand the effect that changing parameters of the assignment policies would have on review quality. The analyses were approved by our Institutional Review Boards.
TPDP. The TPDP workshop received 95 submissions and had a pool of 35 reviewers. Each paper received exactly 3 reviews, and reviewers were assigned 8 or 9 reviews, for a total of 285 assigned reviewer-paper pairs. The reviewers were asked to bid on the papers and could place one of the following bids (the corresponding value of  $B_{i}$  is shown in the parenthesis): "very low" (-1), "low" (-0.5), "neutral" (0), "high" (0.5), or "very high" (1), with "neutral" as the default. The similarity for each reviewer-paper pair was defined as a weighted sum of the bid score,  $B_{i}$ , and text-similarity scores,  $T_{i}$ :  $S_{i} = w_{\mathrm{text}}T_{i} + (1 - w_{\mathrm{text}})B_{i}$ , with  $w_{\mathrm{text}} = 0.5$ . The randomized assignment was run with an upper bound of  $q = 0.5$ . In their review, the reviewers were asked to assess the alignment between the paper and their expertise (between 1: irrelevant and 4: very relevant), and to report their review confidence (between 1: educated guess and 5: absolutely certain). We consider these two responses as our measures of quality.
AAAI. In the AAAI conference, submissions were assigned to reviewers in multiple sequential stages across two rounds of submissions. We examine the stage of the first round where the randomized assignment algorithm was used to assign all submissions to a pool of "senior reviewers." The assignment involved 8,450 papers and 3,145 reviewers; each paper was assigned to one reviewer, and each reviewer was assigned at most 3 or 4 papers based on their primary subject area. The similarity  $S_{i}$  for every reviewer-paper pair  $i$  was based on three scores: text-similarity  $T_{i} \in [0,1]$ , subject-area score  $K_{i} \in [0,1]$ , and bid  $B_{i}$ . Bids were chosen from the following list (with the corresponding value of  $B_{i}$  shown in the parenthesis, where  $\lambda_{\mathrm{bid}} = 1$  is a parameter scaling the impact of positive bids as compared to neutral/negative bids): "not willing" (0.05), "not entered" (1), "in a pinch" (1+0.5λbID), "willing" (1 + 1.5λbID), "eager" (1 + 3λbID). The default option was "not entered". Similarities were computed as:  $S_{i} = (w_{\mathrm{text}}T_{i} + (1 - w_{\mathrm{text}})K_{i})^{1 / B_{i}}$ , with  $w_{\mathrm{text}} = 0.75$ . The actual similarities differed from this base similarity formula in a few special cases (e.g., missing data); we provide the full description of the similarity computation in Appendix F. The randomized assignment was run with  $q = 0.52$ . Reviewers reported an expertise score (between 0: not knowledgeable and 5: expert) and a confidence score (between 0: not confident and 4: very confident), which we consider as our quality measures.
Assumption Suitability. For both the monotonicity and Lipschitz assumptions (as well as the model imputations), we work with the covariates  $X_{i}$ , a vector of the two (TPDP) or three (AAAI) component scores used in the computation of similarities. In Appendix G, we consider whether these assumptions are reasonable with respect to our choice of covariates by evaluating them on the observed outcomes. For the monotonicity assumption, we find that it holds on large majorities of observed outcome pairs. For the Lipschitz assumption, we use a normalized  $\ell_{1}$  distance and choose values of  $L$  corresponding to less than 10%, 5%, and 1% violations on the observed outcomes.
Results. We now present the analyses of the two datasets using the methods introduced in Section 4. For brevity, we report our analysis using self-reported expertise as the quality measure  $Y$ , but include the results using self-reported confidence in Appendix L. When solving the LPs that output alternative randomized assignments (Appendix A), we often encounter multiple unique optimal solutions and employ a persistent arbitrary tie-breaking procedure to choose amongst them (Appendix H).
TPDP. We perform two analyses on the TPDP data, shown in Figure 1 (left). First, we analyze the choice of how to interpolate between the bids and the text similarity when computing the composite similarity score for each reviewer-paper pair. We examine a range of assignments, from an assignment based only on the bids ( $w_{\text{text}} = 0$ ) to an assignment based only on the text similarity ( $w_{\text{text}} = 1$ ), focusing our off-policy evaluation on deterministic assignments (i.e., policies with  $q = 1$ ). Setting  $w_{\text{text}} \in [0, 0.75]$  results in very similar assignments, each of which has Manski bounds overlapping with the on-policy (Figure 1A, left). Within this region, the models (Figure 1B, left), monotonicity bounds (Figure 1C, left), and Lipschitz bounds (Figure 1D, left) all agree that the expertise is similar to the on-policy. However, setting  $w_{\text{text}} \in (0.75, 0.9)$  results in a significant improvement in average expertise, even without any additional assumptions (Figure 1A, left). Finally, setting  $w_{\text{text}} \in (0.9, 1]$  leads to assignments that are significantly different from the assignments supported by the on-policy, which results in many positivity violations and wider confidence intervals, even under the monotonicity and Lipschitz smoothness assumptions (Figures 1A-D, left). Note that within this region, the models significantly disagree on the expertise (Figure 1B, left), indicating that the strong assumptions made by such models may not be accurate. Altogether, these results suggest that putting more weight on the text similarity (versus bids) leads to higher-expertise reviews.
Second, we investigate the "cost of randomization" to prevent fraud, measuring the effect of increasing  $q$  and thereby reducing randomness in the optimized random assignment. We consider values between  $q = 0.4$  and  $q = 1$  (optimal deterministic assignment). Recall the on-policy has  $q = 0.5$ . When varying  $q$ , we find that except for a small increase in the region around  $q = 0.75$ , the average expertise for policies with  $q > 0.5$  is very similar to that of the on-policy (Figures 1A-D, right). These results suggest that using a randomized instead of a deterministic policy does not lead to a significant reduction in self-reported expertise, an observation that should be contrasted with the previously documented reduction in the expected sum-similarity objective under randomized assignment [16]; see further analysis in Appendix I.
AAAI. We perform three analyses on the AAAI data, shown in Figure 1 (right). First, we examine the effect of interpolating between the text-similarity scores and the subject area scores by varying  $w_{\mathrm{text}} \in [0,1]$ , again considering only deterministic policies (i.e.,  $q = 1$ ). The on-policy sets
TPDP
A Manski Bounds
E Manski Bounds
B Model Imputation
F Model Imputation
C Monotonicity Bounds
G Monotonicity Bounds
D Lipschitz Bounds
H Lipschitz Bounds
Figure 1: Expertise of off-policies varying  $w_{\mathrm{text}}$  and  $q$  for TPDP and  $w_{\mathrm{text}}$ ,  $\lambda_{\mathrm{bid}}$ , and  $q$  for AAAI, computed using the different estimation methods described in Section 4. The dashed blue lines indicate Manski bounds around the on-policy expertise, and the grey lines indicate Manski bounds around the off-policy expertise. The error bands (denoted  $\widehat{\mu}_B^{CI}$  for the Manski bounds,  $\widehat{\mu}_{B|M}^{CI}$  for the monotonicity bounds, and  $\widehat{\mu}_{B|L}^{CI}$  for the Lipschitz bounds) represent confidence intervals that asymptotically contain the true value of  $\mu_B$  with probability at least 95%, as described in Appendix C. We observe similar patterns using confidence as an outcome, as shown in Figure 7 in the Appendix. Note that to focus on the most relevant regions of the plots, the vertical axes do not start at zero.
$w_{\mathrm{text}} = 0.75$ . Due to large numbers of positivity violations, the Manski bounds are uninformative (Figure 1E, left), so we turn to the other estimators. The model imputation analysis indicates that policies with  $w_{\mathrm{text}} \geq 0.75$  may have slightly higher expertise than the on-policy and indicates lower expertise in the region where  $w_{\mathrm{text}} \leq 0.5$  (Figure 1F, left). However, the models differ somewhat in their predictions for low  $w_{\mathrm{text}}$ , indicating that the assumptions made by these models may not be reliable. The monotonicity bounds more clearly indicate low expertise compared to the on-policy when  $w_{\mathrm{text}} \leq 0.25$ , but are also slightly more pessimistic about the  $w_{\mathrm{text}} \geq 0.75$  region than the models (Figure 1G, left). The Lipschitz bounds indicate slightly higher than on-policy expertise for  $w_{\mathrm{text}} \geq 0.75$  and potentially suggest slightly lower than on-policy expertise for  $w_{\mathrm{text}} \leq 0.25$  (Figure 1H, left). Overall, all methods of analysis indicate that low values of  $w_{\mathrm{text}}$  result in worse assignments, but the effect of considerably increasing  $w_{\mathrm{text}}$  is unclear.
Second, we examine the effect of increasing the weight on positive bids by varying the values of  $\lambda_{\mathrm{bid}}$ . Recall that  $\lambda_{\mathrm{bid}} = 1$  corresponds to the on-policy and a higher (respectively lower) value of  $\lambda_{\mathrm{bid}}$  indicates greater (respectively lesser) priority given to positive bids relative to neutral/negative bids. We investigate policies that vary  $\lambda_{\mathrm{bid}}$  within the range [0,3], and again consider only deterministic policies (i.e.,  $q = 1$ ). The Manski bounds are again too wide to be informative (Figure 1E, middle). The models all indicate similar values of expertise for all values of  $\lambda_{\mathrm{bid}}$  and are all slightly more optimistic about expertise than the Manski bounds around the on-policy (Figure 1F, middle). The monotonicity and Lipschitz bounds both agree that the  $\lambda_{\mathrm{bid}} \geq 1$  region has slightly higher expertise as compared to the on-policy (Figure 1G-H, middle). Overall, our analyses provide some indication that increasing  $\lambda_{\mathrm{bid}}$  may result in slightly higher levels of expertise.
Finally, we also examine the effect of varying  $q$  within the range [0.4, 1]. Recall that the on-policy sets  $q = 0.52$ . We see that the models (Figure 1F, right), the monotonicity bounds (Figure 1G, right), and the Lipschitz bounds (Figure 1H, right) all strongly agree that the region  $q \geq 0.6$  has slightly higher expertise than the region  $q \in [0.4, 0.6]$ . However, the magnitude of this change is small, indicating that the "cost of randomization" is not very significant.
Power Investigation: Purposefully Bad Policies. As many of the off-policy assignments we consider have relatively similar estimated quality, we also ran additional analyses to show that our methods can discern differences between good policies (optimized toward high reviewer-paper similarity assignments) and policies intentionally chosen to have poor quality ("optimized" toward low reviewer-paper similarity assignments). We refer the reader to Appendix J for further discussion.
In this work, we evaluate the quality of off-policy reviewer-paper assignments in peer review using data from two venues that deployed randomized reviewer assignments. We propose new techniques for partial identification that allow us to draw useful conclusions about the off-policy review quality, even in the presence of large numbers of positivity violations and missing reviews. For a more extensive treatment of the methods proposed in this work, we refer the reader to Khan et al. [45].
Limitations. One limitation of off-policy evaluation is that our ability to make inferences inherently depends on the amount of randomness introduced on-policy. For instance, if there is a small amount of randomness, we will be able to estimate only policies that are relatively close to the on-policy unless we are willing to make some assumptions. The approaches presented in this work allow us to examine the strength of the evidence under a wide range of types and strengths of assumptions (model imputation, boundedness of the outcome, monotonicity, and Lipschitz smoothness) and to test whether these assumptions lead to converging conclusions. We emphasize that these assumptions (except for the boundedness of the outcome) are fundamentally unverifiable in our setting, given that the outcomes for unassigned pairs are unobserved. Thus, our resulting conclusions about the quality of alternative assignments cannot be directly verified without actually deploying the off-policies in question. Additionally, the measurement of review quality is a difficult question for any evaluation method: our analyses use the reviewers' self-reported expertise and confidence scores as proxies for review quality, but the self-reported nature of these measures makes them subject to various biases. Our conclusions should be interpreted in light of the possibility of misalignment between self-reported expertise/confidence and the true review quality. Finally, our substantive conclusions are based on analyses of data from two venues, and thus further work is needed to test their generalizability.