SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Specializing this result to the definitions  $\bar{D}_0$  and  $\tilde{D}_0$  (and, in particular, the definition of  $\varepsilon_{n,L}^{*}$ ) completes the proof.
Proof of Lemma 4.
Our proof will proceed via a covering argument. In particular, we cover the feature space  $[0,1]^d$  with a set of non-overlapping  $d$ -dimensional hypercubes, each of which has edge length  $0 < b < 1$ , and show that sufficiently many pairs  $(x,x')$  must lie in the same 'small' hypercube. To that end, let  $C = \{c_1 \ldots c_k\}$  be a set of hypercubes of edge length  $b$  with the following properties:
$$
\forall c \in C, c \subseteq [ - b, 1 + b ] ^ {d} \tag {39}
$$
$$
\forall c, c ^ {\prime} \in C, c \cap c ^ {\prime} = \emptyset \tag {40}
$$
$$
\forall x \in D _ {0}, \exists c \in C \mid x \in c \tag {41}
$$
Where  $D_0$  is the observed data. It's clear that such a covering  $C$  must exist, for example by arranging  $c_1 \ldots c_k$  in a regularly spaced grid which cover  $[0, 1]^d$  (though note that per condition (39), some of these 'small' hypercubes may extend outside  $[0, 1]^d$  if  $b$  does not evenly divide 1). Such a covering may be difficult to index as care must be exercised around the boundaries of each small hypercube; however, as we only require the existence of such a covering, we ignore these details. We now state the following elementary facts:
$$
| C | \leq \left\lfloor \frac {(1 + 2 b) ^ {d}}{b ^ {d}} \right\rfloor \tag {42}
$$
$$
\forall c \in C, x, x ^ {\prime} \in c, \| x - x ^ {\prime} \| _ {2} \leq b \sqrt {d} \tag {43}
$$
Where (42) follows because the volume of each  $c \in C$  is  $b^{d}$ , and the total volume of all such hypercubes cannot exceed the volume of the containing hypercube  $[-b, 1 + b]^{d}$ , which gives us an upper bound on the size of the cover  $C$ . Furthermore, (43) tells us that for any  $(x, x')$  which lie in the same 'small' hypercube  $c$ , we have  $\| x - x' \|_{2} \leq b\sqrt{d}$ .
Let  $n_c \coloneqq |\{x_i \mid x_i \in c\}|$  denote the number of observations contained in each small hypercube  $c \in C$ .
Corollary 5.2 For any  $c \in C$ , there exist at least  $\left\lfloor \frac{n_c}{2} \right\rfloor$  disjoint pairs  $(x, x') \in c$  such that  $\|x - x'\|_2 \leq b\sqrt{d}$ .
With these preliminaries in place, we'll proceed to prove lemma 4. To do this, we'll first state one additional auxiliary lemma.
Let  $N_{a,b} \coloneqq \frac{a^d}{b^d} \geq \left\lfloor \frac{a^d}{b^d} \right\rfloor$ , an upper bound on the number of non-overlapping 'small' hypercubes with edge length  $b$  which can fit into  $[0, a]^d$ . We'll show for any  $z > 0$ , with  $b \coloneqq \frac{z}{\sqrt{d}}$ ,  $a \coloneqq 1 + 2b$ , we have:
Lemma 6 (Pairwise distance in terms of packing number)
$$
n \geq 2 N _ {a, b} \Rightarrow \exists \frac {n}{4} \text {p a i r s s a t i s f y i n g} \| x - x ^ {\prime} \| _ {2} \leq z \tag {44}
$$
That is, the pairwise distance between the closest set of  $\frac{n}{4}$  pairs (half the observed data in total) can be written in terms of the appropriately parameterized covering number. We defer the proof of this lemma to the following section. For now, we simply plug in the definition of  $N_{a,b}$  and rearrange to recover:
$$
n \geq 2 N _ {a, b} = 2 \frac {\left(1 + 2 \frac {z}{\sqrt {d}}\right) ^ {d}}{\left(\frac {z}{\sqrt {d}}\right) ^ {d}} \Rightarrow \frac {2 ^ {\frac {1}{d}} \sqrt {d}}{n ^ {\frac {1}{d}} - 2 ^ {1 + \frac {1}{d}}} \leq z \tag {45}
$$
Recall that  $z$  is the maximum distance between any pairs  $(x, x')$  contained in the same small hypercube with edge length  $\frac{z}{\sqrt{d}}$ . The preceding argument holds for all  $z > 0$  which satisfy (45), so in particular, it holds for
$$
z ^ {*} := \frac {2 ^ {\frac {1}{d}} \sqrt {d}}{n ^ {\frac {1}{d}} - 2 ^ {1 + \frac {1}{d}}}. \tag {46}
$$
$z^{*}$  is the maximum pairwise distance corresponding to one possible matching on  $\frac{n}{4} (x, x')$  pairs, so this further implies that there exists a matching  $M$  of size  $\frac{n}{4}$  such that:
$$
\max _ {(x, x ^ {\prime}) \in M} | | x - x ^ {\prime} | | _ {2} \leq \frac {2 ^ {\frac {1}{d}} \sqrt {d}}{n ^ {\frac {1}{d}} - 2 ^ {1 + \frac {1}{d}}} = O (n ^ {- \frac {1}{d}})
$$
With probability 1. Thus, it follows that the maximum distance between any pair in the optimal matching  $d_{\frac{n}{4}}^*$  also satisfies:
$$
d _ {\frac {n}{4}} ^ {*} = O \left(\frac {2 ^ {\frac {1}{d}} \sqrt {d}}{n ^ {\frac {1}{d}} - 2 ^ {1 + \frac {1}{d}}}\right) = O \left(n ^ {- \frac {1}{d}}\right)
$$
With probability 1, as desired. This establishes the existence of a matching of up to  $L = \frac{n}{4}$  disjoint pairs  $(x, x') \in [0, 1]^d$  such that the maximum distance between any such pair scales like  $O\left(n^{-\frac{1}{d}}\right)$ .
We also consider the case where instead of  $\mathcal{X} \coloneqq [0,1]^d$ , we instead have  $\mathbb{P}\left(X \in [0,1]^d\right) \geq 1 - \delta$  for some  $\delta \in (0,1)$ . For example, this will capture the case where  $X$  is a (appropriately re-centered and re-scaled) multivariate Gaussian. In this case, we provide a corresponding high probability version of lemma 4.
Corollary 6.1 Suppose instead of  $\mathcal{X} \coloneqq [0,1]^d$ , we have for some  $\delta \in (0,1)$ :
$$
\mathbb {P} (X \in [ 0, 1 ] ^ {d}) \geq 1 - \delta \tag {47}
$$
Define  $m\coloneqq (1 - \delta)^2 n$
We can then show:
$$
\mathbb {P} \left(d _ {\frac {m}{4}} ^ {*} \leq \frac {2 ^ {\frac {1}{d}} \sqrt {d}}{m ^ {\frac {1}{d}} - 2 ^ {1 + \frac {1}{d}}}\right) \geq 1 - e ^ {- \frac {\delta^ {2} (1 - \delta) n}{2}} \tag {48}
$$
That is, we can still achieve a constant factor approximation to the optimal matching in Lemma 4 with probability that exponentially approaches 1.
Proof of Corollary 6.1
Define the set of points which falls in  $[0,1]^d$  as follows:
$$
S _ {0} := \left\{X _ {i} \mid X _ {i} \in [ 0, 1 ] ^ {d} \right\} \tag {49}
$$
$$
n _ {0} := \left| S _ {0} \right| \tag {50}
$$
It is clear that in this setting, the proof of lemma 4 holds if we simply replace  $n$  with  $n_0$ , the realized number of observations which fall in  $[0,1]^d$ . However,  $n_0$  is now a random quantity which follows a binomial distribution with mean  $(1 - \delta)n$  (recall that we assume  $(x_i,y_i,\hat{y}_i)$  are drawn i.i.d. throughout). Thus, all that remains is to bound  $n_0$  away from 0, which we can do via a simple Chernoff bound:
$$
\mathbb {P} \left(n _ {0} \leq (1 - \delta) ^ {2} n\right) \leq e ^ {- \frac {\delta^ {2} (1 - \delta) n}{2}} \tag {51}
$$
$$
\mathbb {P} \left(n _ {0} \geq (1 - \delta) ^ {2} n\right) \geq 1 - e ^ {- \frac {\delta^ {2} (1 - \delta) n}{2}} \tag {52}
$$
Thus, we have shown  $n_0 \geq m$  with the desired probability. It is clear that we only require a lower bound on  $n_0$  to recover the result of Theorem 4, as additional observations which fall in  $[0,1]^d$  can only improve the quality of the optimal matching  $d_{\frac{m}{4}}^*$ .
Proof of Lemma 5
We will show that the procedure in ExpertTest which greedily pairs the closest remaining pair of points  $L$  times will always be able to choose at least one of the pairs in an optimal matching of size  $2L$ . Intuitively, this is because each pair  $(x, x')$  chosen by ExpertTest can only 'rule out' at most two pairs  $(x, x''), (x', x''')$  in any optimal matching of size  $2L$ . Thus, our greedy algorithm for choosing  $L$  pairs can perform no worse than an optimal matching of size  $2L$ , the sense of minimizing the maximum pairwise distance.
Let  $m_{2L}^{*}$  be an optimal matching of size  $2L$  in the sense of (26). Then suppose towards contradiction that:
$$
\max  _ {l \in [ L ]} \| x _ {2 l - 1} - x _ {2 l} \| _ {2} > d _ {2 L} ^ {*} \tag {53}
$$
Where  $d_{2L}^{*}$  is the smallest achievable maximum distance for any matching of size  $2L$  as in (27).
Finally, let  $l_{m} \coloneqq \arg \min_{l \in [L]} ||x_{2l-1} - x_{2l}||_{2} > d_{2L}^{*}$ ; i.e. the first pair which is chosen by ExpertTest that violates (53). Because pairs are chosen greedily to minimize  $\ell_{2}$  distance, and  $m_{2L}^{*}$  is a matching of size  $2L$  where all pairs are separated by at most  $d_{2L}^{*}$  under the  $\ell_{2}$  norm, it must be that none of the pairs which make up  $m_{2L}^{*}$  were available to ExpertTest at the  $l_{m}$ -th iteration. In particular, at least one element of every  $(x, x')$  pair in  $m_{2L}^{*}$  must have been selected on a previous iteration:
$$
\forall (x, x ^ {\prime}) \in m _ {2 L} ^ {*}, x \in \{x _ {1} \dots x _ {2 l _ {m} - 2} \} \vee x ^ {\prime} \in \{x _ {1} \dots x _ {2 l _ {m} - 2} \} \tag {54}
$$
As  $m_{2L}^{*}$  contains  $2L$  disjoint pairs  $-4L$  observations total – this implies that  $2l_{m} - 2 \geq 2L \Rightarrow l_{m} - 1 \geq L \Rightarrow l_{m} > L$ . This is a contradiction, as ExpertTest only chooses  $L$  pairs, so  $l_{m}$  only ranges in  $[1, L]$ . This completes the proof.
Corollary 6.2 Validity in finite samples
Theorem 2 implies that we can achieve a bound on the excess type one error in finite samples if we knew the constant  $C$  in (12). In particular, let
$$
m ^ {*} := \max  _ {\ell \in [ L ]} \left\| x _ {2 \ell - 1} - x _ {2 \ell} \right\| _ {2} \tag {55}
$$
$$
\epsilon^ {*} := \max  _ {r \in \left[ (1 + C m ^ {*}) ^ {- 2}, (1 + C m ^ {*}) ^ {2} \right]} \left| \frac {1}{r + 1} - \frac {1}{2} \right| \tag {56}
$$
Then (10) implies that we can always construct a valid (if underpowered) test at exactly the nominal size  $\alpha$  by updating our REJECT threshold to
$$
\alpha - \left(1 - (1 - \epsilon^ {*}) ^ {L}\right) - \frac {1}{K + 1}
$$
Proof of lemma 6
let  $C := \{c_1 \ldots c_k\}$  denote any set of  $k$  'small' nonoverlapping hypercubes of edge length  $b$  satisfying properties (39), (40) and (41). As discussed in the proof of lemma 4, each element of  $C$  is not guaranteed to lie strictly in  $[0,1]$ . Rather, each  $c \in C$  must merely intersect  $[0,1]^d$ , implying that each element of the cover is instead contained in the slightly larger hypercube  $[-b, 1 + b]^d$ . As in the proof of lemma 4, we'll again let  $n_c$  denote the number of observations  $x_i$  which lie in some  $c \in C$ .
By Corollary 5.2, we have that  $\left\lfloor \frac{n_c}{2} \right\rfloor$  pairs in each  $c \in C$  will satisfy  $||x - x'||_2 \leq b\sqrt{d} = z$ . Thus what's left to show is that:
$$
n \geq 2 N _ {a, b} \Rightarrow \sum_ {j \in [ k ]} \lfloor \frac {n _ {c _ {j}}}{2} \rfloor \geq \frac {n}{4}
$$
We can see this via the following argument:
$$
\sum_ {j \in [ k ]} \left\lfloor \frac {n _ {c _ {j}}}{2} \right\rfloor \geq \sum_ {j \in [ k ]} \left(\frac {n _ {c _ {j}}}{2} - \frac {1}{2}\right) \tag {57}
$$
$$
= \frac {n}{2} - \frac {k}{2} \tag {58}
$$
$$
\geq \frac {n}{2} - \frac {N _ {a , b}}{2} \tag {59}
$$
$$
\geq \frac {n}{2} - \frac {n}{4} = \frac {n}{4} \tag {60}
$$
Where (59) follows from (42) and the definition of  $N_{a,b}$ , and (60) follows because  $n \geq 2N_{a,b}$  by assumption. This completes the proof.
E Omitted Details from Section 5
E.1 Identifying relevant patient encounters and classifying outcomes
As described in Section 5, we consider a set of 3617 patients who presented with signs or symptoms of acute gastrointestinal bleeding at the emergency department at a large quaternary academic hospital system from January 2014 to December 2018. These patient encounters were identified using a database mapping with a standardized ontology (SNOMED-CT) and verified by manual physician chart review. Criteria for inclusion were the following: any text that identifies acute gastrointestinal bleeding for hematemesis, melena, hematochezia from either patient report or physical exam findings
(which were considered equally valid for the purposes of inclusion). Exclusion criteria were the following: patients with other reasons for overt bleeding symptoms (e.g. epistaxis) or missingness in input variables required to calculate the Glasgow-Blatchford Score. As mentioned in Section 5, only upper gastrointestinal bleeding guidelines recommend the Glasgow-Blatchford Score for routine clinical use. However, the Glasgow-Blatchford Score has also been validated for use in patients with lower gastrointestinal bleeding, and in our setting the GBS was applied to patients who presented with signs of symptoms of either upper or lower gastrointestinal bleeding. We refer the reader to Section 5 for additional details and references.
This identified a set of 3627 patients, of which a further 10 were removed from consideration due to unclear emergency department disposition (neither Admit nor Discharge). As described in Section 5, we record an adverse outcome  $(Y = 1)$  for admitted patients who required some form of hemostatic intervention (excluding a diagnostic endoscopy or colonoscopy), or patients who are readmitted or die within 30 days. We record an outcome of 0 for all other patients.
The use of readmission as part of the adverse event definition is subject to two important caveats. First, we are only able to observe patients who are readmitted within the same hospital system. Thus, although the hospital system we consider is the dominant regional health care network, it is possible that some patients subsequently presented elsewhere with signs or symptoms of AGIB; such patients would be incorrectly classified as not having suffered an adverse outcome. Second, we only record an outcome of 1 for patients who are readmitted with signs or symptoms of AGIB, subject to the same inclusion criteria defined above. Patients who are readmitted for other reasons are not recorded as having suffered an adverse outcome.
E.2 The special case of binary outcomes and predictions
In our experiments we define the loss measure  $F(D) \coloneqq \frac{1}{n} \sum_{i} \mathbb{1}[y_i \neq \hat{y}_i]$ , but it's worth remarking that this is merely one choice within a large class of natural loss functions for which ExpertTest produces identical results when  $Y, \hat{Y}$  are binary. In particular, observe that a swap of  $(y_1, \hat{y}_1)$ ,  $(y_2, \hat{y}_2)$  can only change the value of  $F(\cdot)$  if  $y_1 \neq y_2$  and  $\hat{y}_1 \neq \hat{y}_2$  (we'll assume throughout that all observations contribute equally to the loss; i.e. it is invariant to permutations of the indices  $i \in [n]$ ). This implies that there are only  $2^2$  out of  $2^4$  possible configurations of  $(y_1, \hat{y}_1, y_2, \hat{y}_2)$  where a swap can change the loss at all. Of these, two configurations create a false negative and a false positive in the synthetic data which did not exist in the observed data:
$$
\underbrace {\left(y _ {1} = 1 , \hat {y} _ {1} = 1 , y _ {2} = 0 , \hat {y} _ {2} = 0\right)} _ {\text {o r i g i n a l d a t a}} \xrightarrow {} \underbrace {\left(y _ {1} = 1 , \hat {y} _ {1} = 0 , y _ {2} = 0 , \hat {y} _ {2} = 1\right)} _ {\text {s y n t h e t i c d a t a}}
$$
$$
(y _ {1} = 0, \hat {y} _ {1} = 0, y _ {2} = 1, \hat {y} _ {2} = 1) \underset {\text {s w a p}} {\rightarrow} (y _ {1} = 0, \hat {y} _ {1} = 1, y _ {2} = 1, \hat {y} _ {2} = 0)
$$
The other two configurations which change the loss are symmetric, in that a swap removes both a false negative and false positive that exists in the observed data:
$$
\left(y _ {1} = 0, \hat {y} _ {1} = 1, y _ {2} = 1, \hat {y} _ {2} = 0\right) \underset {\text {s w a p}} {\rightarrow} \left(y _ {1} = 0, \hat {y} _ {1} = 0, y _ {2} = 1, \hat {y} _ {2} = 1\right)
$$
$$
\left(y _ {1} = 1, \hat {y} _ {1} = 0, y _ {2} = 0, \hat {y} _ {2} = 1\right) \underset {\text {s w a p}} {\rightarrow} \left(y _ {1} = 1, \hat {y} _ {1} = 1, y _ {2} = 0, \hat {y} _ {2} = 0\right)
$$
Thus, for any natural loss function which is strictly increasing in the number of mistakes  $\sum_{i}\mathbb{1}[y_i\neq \hat{y}_i]$ , the first two configurations of  $(y_{1},\hat{y}_{1},y_{2},\hat{y}_{2})$  will induce swaps which strictly increase the loss, while the latter two will induce swaps that strictly decrease the loss. This means that for a given set of  $L$  pairs, we can compute the number of swaps which would increase (respectively, decrease) the loss for any function in this class of natural losses. In particular, this class includes loss functions which may assign arbitrarily different costs to false negatives and false positives. Thus, in the particular context of assessing physician triage decisions, our results are robust to variation in the way different physicians, patients or other stakeholders might weigh the relative cost of false negatives (failing to hospitalize patients who should have been admitted) and false positives (hospitalizing patients who could have been discharged to outpatient care).
E.3 Alternative feature spaces
In Section 5, we run ExpertTest by pairing patients who appear as close as possible (and, in most cases, identical) with respect to the nine features which are provided as input to the Glasgow-Blatchford score. While this test has perhaps the most natural interpretation in our setting, we could in general choose other ways of representing patients. In particular, a natural question to ask is whether physicians successfully distinguish patients who present with identical Glasgow-Blatchford scores (or, more generally, with identical predictions under some model of interest). That is, rather than letting  $X \in [0,1]^{9}$  be the nine patient characteristics which are provided as input to the GBS, we instead let  $X \in \{0,1 \ldots 23\}$  be the GBS itself. While this is in some sense 'weaker' than the test we run in Section 5 – the GBS is a deterministic function of the richer feature space we chose there – recall that the validity of ExpertTest relies on finding pairs which are identical or nearly identical in the feature space. Thus, choosing to instead condition on a simple univariate prediction can in general yield useful insight when ExpertTest fails to offer conclusive results in the 'natural' input space. We present the results of this experiment in Table 3 below.
Table 3: The results of running ExpertTest, where each pair of patients is chosen to be as similar as possible with respect to their Glasgow-Blatchford scores.  $L$  indicates the number of pairs selected for the test, of which 'mismatched pairs' are not identical to each other. Swaps that decrease (respectively, increase) loss indicates how many of the  $L$  pairs result in a decrease (respectively, increase) in the 0/1 loss when their corresponding hospitalization decisions are exchanged with each other.  $\tau$  is the p-value obtained from running ExpertTest.
As we can see, we get very similar results to those presented in Section 5. Consistent with Table 2, we find no mismatched pairs for  $L \leq 1000$ , and we again obtain the smallest possible p-values of  $\frac{1}{K + 1} = \frac{1}{1001}$  for larger values of  $L$ .
Another experiment we might run is to condition on a richer feature space than the one we chose in Section 5. In particular, recall that of the nine inputs to the GBS, four of them (blood urea nitrogen (BUN), hemoglobin (HGB), systolic blood pressure (SBP) and pulse) are real-valued test scores or vital signs. Part of the standard construction of the Glasgow-Blatchford Score entails converting these features to simple discrete scales, with larger values indicating that a patient is higher risk. A useful consequence of this convention is that it allows us to find large numbers of patients who appear identical with respect to these discretized features. However, we can also run ExpertTest in the 'original' feature space, where each of these four characteristics take values in  $\mathbb{R}$  and thus effectively guarantee that we fail to find pairs of patients who are exactly identical. We present the results of this test (where the only transformation we apply is to again rescale each feature to lie in [0, 1]) in Table 4 below.
Here we again see strong evidence of expertise, with p-values equal to  $\frac{1}{K + 1} = \frac{1}{1001}$  for larger values of  $L$ . However, unlike in Table 2 and Table 3, we now fail to find identical pairs of patients for any value of  $L$ . This raises the possibility that ExpertTest is not valid in this setting, as type I error control now relies on the value of  $\varepsilon_{n,L}^{*}$  (11). While we cannot compute  $\varepsilon_{n,L}^{*}$  without making assumptions about the underlying data distribution, we can provide a sense check by examining the Euclidean distance between each pair of patients. If it is very close to 0 for most or all pairs, we might reasonably expect that ExpertTest recovers approximate type I error control. We present these results in Figure 1 below.
As expected, we fail to find any pairs of patients who are identical in this partially real-valued feature space. Nonetheless, we see that for  $L \leq 1000$ , every pair is chosen such that the Euclidean distance between them is very close to 0, which suggests that ExpertTest is likely to be approximately valid for these experiments. We also see however that for  $L = 1808$ , some pairs are chosen such that the
Table 4: The results of running ExpertTest, where each pair of patients is chosen to be as similar as possible with respect to nine patient characteristics. The Glasgow-Blatchford score is computed from these nine characteristics, after a pre-processing step which discretizes each feature.  $L$  indicates the number of pairs selected for the test, of which 'mismatched pairs' are not identical to each other. Swaps that decrease (respectively, increase) loss indicates how many of the  $L$  pairs result in a decrease (respectively, increase) in the 0/1 loss when their corresponding hospitalization decisions are exchanged with each other.  $\tau$  is the p-value obtained from running ExpertTest.
pairwise distance is more than  $30\%$  of its maximum possible value, suggesting that ExpertTest does indeed incur substantial excess type I error when  $L = 1808$ .
F Numerical Experiments
We first elaborate here on the example 1 presented in the introduction. Consider the following stylized data generating process:
Example: experts can add value despite poor performance.
Let  $X, U, \epsilon_1, \epsilon_2$  be independent random variables distributed as follows:
$$
X \sim \mathcal {U} ([ - 2, 2 ]), U \sim \mathcal {U} ([ - 1, 1 ]), \epsilon_ {1} \sim \mathcal {N} (0, 1), \epsilon_ {2} \sim \mathcal {N} (0, 1)
$$
Where  $\mathcal{U}(\cdot)$  and  $\mathcal{N}(\cdot, \cdot)$  are the uniform and normal distribution, respectively. Suppose the true data generating process for the outcome of interest  $Y$  is
$$
Y = X + U + \epsilon_ {1}
$$
Suppose a human expert constructs a prediction  $\hat{Y}$  which is intended to forecast  $Y$  and can be modeled as:
$$
\hat {Y} = \operatorname {s i g n} (X) + \operatorname {s i g n} (U) + \epsilon_ {2}
$$
Where  $\operatorname{sign}(X) \coloneqq \mathbb{1}[X > 0] - \mathbb{1}[X < 0]$ .
We compare this human prediction to that of an algorithm  $\hat{f}(\cdot)$  which can only observe  $X$ , and correctly estimates
$$
\hat {f} (X) = \mathbb {E} [ Y \mid X ] = X
$$
As described in the introduction, we use this example to demonstrate that ExpertTest can detect that the forecast  $\hat{Y}$  is incorporating the unobserved  $U$  even though the accuracy of  $\hat{Y}$  is substantially worse than that of  $\hat{f}(X)$ . In particular, we consider the mean squared error (MSE) of each of these predictors:
Algorithm MSE  $\coloneqq \frac{1}{n}\sum_{i}(Y_{i} - \hat{f} (X_{i}))^{2}$
Human MSE  $\coloneqq \frac{1}{n}\sum_{i}(Y_{i} - \hat{Y}_{i})^{2}$