SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We'll show below that the Algorithm MSE is substantially smaller than the Human MSE. However, we may also wonder whether the performance of the human forecast  $\hat{Y}$  is somehow artificially constrained by the relative scale of  $\hat{Y}$  and  $Y$ , as the  $\mathrm{sign}(\cdot)$  operation restricts the range of  $\hat{Y}$ . For example, a forecaster who always outputs  $\hat{Y} = \frac{Y}{100}$  is perfectly correlated with the outcome but will incur very large squared error; this is a special case of the more general setting where human forecasts are directionally correct but poorly calibrated. To test this hypothesis, we can run ordinary least squares regression (OLS) of  $Y$  on  $\hat{Y}$  and compute the squared error of this rescaled prediction. It is well known OLS estimates the optimal linear rescaling with respect to squared error, and we further use the in sample MSE of this rescaled prediction to provide a lower bound on the achievable loss. In particular, let:
$$
\left(\beta^ {*}, c ^ {*}\right) := \min  _ {\beta , c \in \mathbb {R}} \| Y - \beta \hat {Y} - c \| _ {2} ^ {2} \tag {61}
$$
$$
\text {R e s c a l e d H u m a n M S E} := \frac {1}{n} \sum_ {i} \left(Y _ {i} - \beta^ {*} \hat {Y} _ {i} - c ^ {*}\right) ^ {2} \tag {62}
$$
In Table 5 we report the mean squared error (plus/minus two standard deviations) over 100 draws of  $n = 1000$  samples from the data generating process described above. As we can see, both the original and rescaled human forecasts substantially underperform  $\hat{f}(\cdot)$ .
We now assess the power of ExpertTest in this setting by repeatedly simulating  $n = 1000$  draws of  $(X, U, \epsilon_1, \epsilon_2)$  along with the associated outcomes  $Y := X + U + \epsilon_1$  and expert predictions  $\hat{Y} := \mathrm{sign}(X) + \mathrm{sign}(U) + \epsilon_2$ . We sample 100 datasets in this manner, and run ExpertTest on each one with  $L, K = 100$ , and the distance metric  $m(x, x') := \sqrt{(x - x')^2}$ . The distribution of p-values  $\tau_1 \ldots \tau_{100}$  is plotted in Figure 2.
We see that ExpertTest produces a highly nonuniform distribution of the p-value  $\tau$ , and rejects the null hypothesis  $94\%$  of the time at a critical value of  $\alpha = .05$ . To assess whether this power comes at the expense of an inflated type I error, we also run ExpertTest with both  $X$  and  $U$  'observed'; in particular, suppose the distance measure was instead  $m((x,u),(x',u')) = \sqrt{(x - x')^2 + (u - u')^2}$  with everything else defined as above. The distribution of  $\tau$  in this setting is again plotted in Figure 3.
When both  $X$  and  $U$  are observed, and thus the null hypothesis should not be rejected, we instead see that we instead get an approximately uniform distribution of  $\tau$  with a false discovery rate of only .03 at a critical value of  $\alpha = .05$ . Thus, the power of ExpertTest to detect that the synthetic expert is incorporating some unobserved information  $U$  does not come at the expense of inflated type I error, at least in this synthetic example.
Assessing the power of ExpertTest
We now present additional simulations to highlight how the power of ExpertTest scales with the number of pairs  $L$  and the sample size  $n$  in a more general setting. In particular, we consider a simple
synthetic dataset  $(x_{i},y_{i},\hat{y}_{i}),i\in [n]\equiv \{1,\ldots ,n\}$  where  $x_{1}\dots x_{n} = [1,1,2,2,\dots \frac{n}{2},\frac{n}{2}]'$  and  $y_{1}\dots y_{n}$  is the alternating binary string  $[0,1,0,1\dots 0,1]^{\prime}$  (we consider only even  $n$  for simplicity). This guarantees that each of the  $L$  pairs chosen are such that  $(x_{2\ell -1} = x_{2\ell})$  and  $y_{2\ell -1}\neq y_{2\ell}$ . Importantly, it's also clear that  $x$  is uninformative about the true outcome  $y$  - if the expert can perform better than random guessing, it must be by incorporating some unobserved signal  $U$ .
We model this unobserved signal by an 'expertise parameter'  $\delta \in [0, \frac{1}{2}]$ . In particular, for each pair  $(y_{2\ell - 1}, y_{2\ell})$  for  $\ell \in [1 \ldots \frac{n}{2}]$ , we sample  $(\hat{y}_{2\ell - 1}, \hat{y}_{2\ell})$  such that  $(\hat{y}_{2\ell - 1}, \hat{y}_{2\ell}) = (y_{2\ell - 1}, y_{2\ell})$  with probability  $\frac{1}{2} + \delta$  and  $(y_{2\ell}, y_{2\ell - 1})$  otherwise. Intuitively,  $\delta$  governs the degree to which the expert predictions  $\hat{Y}$  incorporate unobserved information – at  $\delta = 0$ , we model an expert who is randomly guessing, whereas at  $\delta = \frac{1}{2}$  the expert predicts the outcome with perfect accuracy.
First, we consider the case of  $n \in \{200, 600, 1200\}$  and fix  $L$  at  $\frac{n}{8}$  as suggested by the proof of Theorem 2. For each of these cases, we examine how the discovery rate scales with the expertise parameter  $\delta \in \{0, .05, .45, .50\}$ . In particular, we choose a critical threshold of  $\alpha = .05$  and compute how frequently ExpertTest rejects  $H_0$  over 100 independent draws of the data for each value of  $\delta$ . These results are plotted below in Figure 4.
Unsurprisingly, the power of ExpertTest depends critically on the sample size - at  $n = 1200$ , ExpertTest achieves  $80\%$  power in rejecting  $H_0$  when the expert only performs modestly better than random guessing ( $\delta \approx .1$ ). In contrast, at  $n = 200$ , ExpertTest fails to achieve  $80\%$  power until  $\delta \approx .25$  - corresponding to an expert who provides the correct predictions over  $75\%$  of the time even when the observed  $x$  is completely uninformative about the true outcome.
Next we examine how the power of ExpertTest scales with  $L$ . We now fix  $n = 600$  and let  $\delta = .2$  to model an expert who performs substantially better than random guessing, but is still far from providing perfect accuracy. We then vary  $L \in \{20, 40 \dots 200\}$  and plot the discovery rate (again at a critical value of  $\alpha = .05$ , over 500 independent draws of the data) for each choice of  $L$ . These results are presented below in Figure 5.
As expected, we see that power is monotonically increasing in  $L$ , and asymptotically approaching 1. With  $\delta = .2$ , we see that ExpertTest achieves power in the neighborhood of only  $50\%$  with  $L = 20$  pairs, but sharply improves to approximately  $80\%$  power once  $L$  increases to 40. Beyond this threshold we see that there are quickly diminishing returns to increasing  $L$ .
Excess type I error of ExpertTest
Recall that, per Theorem 1, ExpertTest becomes more likely to incorrectly reject  $H_0$  as  $L$  increases relative to  $n$ . In particular, larger values of  $L$  will force ExpertTest to choose  $(x, x')$  pairs which are farther apart under any distance metric  $m(\cdot, \cdot)$ , and thus induce larger values of  $\varepsilon_{n,L}^*$  as defined in
(11). Furthermore, even for fixed  $\varepsilon_{n,L}^{*} > 0$ , the type one error bound given in Theorem 1 degrades with  $L$ . We empirically investigate this phenomenon via the following numerical simulation.
First, let  $X = (X_{1},X_{2},X_{3})\subset \mathbb{R}^{3}$  be uniformly distributed over  $[0,10]^3$ . Let  $Y = X_{1} + X_{2} + X_{3} + \epsilon_{1}$  and  $\hat{Y} = X_{1} + X_{2} + X_{3} + \epsilon_{2}$ , where  $\epsilon_1,\epsilon_2$  are independent standard normal random variables. In this setting, it's clear that  $H_0:Y\perp \hat{Y}\mid X$  holds.
We repeatedly sample  $n = 500$  independent observations from this distribution over  $(X, Y, \hat{Y})$  and run ExpertTest for each  $L \in \{25, 50 \dots 250\}$ . We let  $K = 50$  and  $m(x, x') \coloneqq ||x - x'||_2^2$  be the  $\ell_2$  distance. We let the loss function  $F(\cdot)$  be the mean squared error of  $\hat{Y}$  with respect to  $Y$ . For each scenario we again choose a critical threshold of  $\alpha = .05$ , and report how frequently ExpertTest incorrectly rejects the null hypothesis over 50 independent simulations in Figure 6.
As we can see, the type I error increases sharply as a function of  $L$ , and ExpertTest incurs a false discovery rate of  $100\%$  at the largest possible value of  $L = \frac{n}{2}$ . This suggests that significant care should be exercised when choosing the value of  $L$ , particularly in small samples, and responsible use of ExpertTest will involve leveraging domain expertise to assess whether the pairs chosen are indeed 'similar' enough to provide type I error control.
G Pseudocode for ExpertTest
In this section we provide pseudocode for ExpertTest. Inputs  $D_0, L, K, \alpha, F(\cdot), m(\cdot, \cdot)$  are as defined in Section 3.