SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We remark briefly on the role of the parameters  $L$  and  $K$  in this result. To begin with,  $\frac{1}{K + 1}$  is embedded in the type I error, and thus taking the number of resampled datasets  $K$  to be as large as possible (subject only to computational constraints) sharpens the validity guarantee. We also observe that the bound becomes weaker as  $L$  increases. However, observe also that ExpertTest is implicitly using an  $L$ -dimensional distribution (or  $L$  fresh samples) to reject  $H_0$ , which means that increasing  $L$  also provides additional power to the test.
Notice also that the odds ratio (9) is guaranteed to be 1 if  $x = x'$ , regardless of the underlying distribution  $\mathcal{D}$ . This is not a coincidence, and our test is based implicitly on the heuristic that the odds ratio will tend away from 1 as the distance  $m(x,x')$  increases (we quantify this intuition precisely below). Thus, increasing  $L$  will typically also increase  $\varepsilon_{n,L}^{*}$ , because larger values of  $L$  will force us to pair additional observations  $(x,x')$  which are farther apart under the distance metric.
The type one error bound (10) suggests that we can balance the trade off between validity and power when  $\varepsilon_{n,L}^{*}L\ll 1$  or  $o(1)$ , as the right hand side of (10) reduces to  $\alpha +o(1)$ . Next we describe a representative generative setup where there is a natural choice of  $L$  that leads to  $\varepsilon_{n,L}^{*}L = o(1)$ .
Generative model. Let  $\mathcal{X} = [0,1]^d\subset \mathbb{R}^d$ . Let the conditional density of the human expert's forecasts  $Q(\cdot |x)$  be smooth. Specifically, for any  $x,x^{\prime}\in [0,1]^{d}$ ,
$$
\sup  _ {\hat {y} \in \mathbb {R}} \frac {Q (\hat {y} \mid X = x)}{Q (\hat {y} \mid X = x ^ {\prime})} \leq 1 + C \times \| x - x ^ {\prime} \| _ {2}, \tag {12}
$$
for some constant  $C > 0$ . Under this setup, Theorem 1 reduces to the following.
Theorem 2 (Asymptotic Validity) Given  $\alpha \in (0,1)$  and under (12), with the appropriate choice of  $L\geq 1$ , the type I error of ExpertTest satisfies
$$
\mathbb {P} \left(\tau_ {K} \leq \alpha\right) \leq \alpha + o (1). \tag {13}
$$
as  $n,K\to \infty$
Intuitively, (12) is intended to model a forecasting rule which is 'simple,' in the sense that human experts don't finely distinguish between instances whose feature vectors are close under the  $\ell_2$  norm. Importantly, this does not rule out the possibility that predictions for two specific  $(x,x^{\prime})$  instances could differ substantially - only that the distributions  $\hat{Y}\mid X = x$  and  $\hat{Y}\mid X = x^{\prime}$  are similar when  $x\approx x^{\prime}$ . We make no such assumption about  $\mathcal{D}_{Y|X}$ , the conditional distribution of the true outcomes.
Proofs of theorems 1 and 2 can be found in Appendices B and C respectively. We now illustrate the utility of our test with an empirical study of physician hospitalization decisions.
5 A case study: physician expertise in emergency room triage
Emergency room triage decisions present a natural real-world setting for our work, as we can assess whether physicians make hospitalization decisions by incorporating information which is not available to an algorithmic risk score. We consider the particular case of patients who present in the emergency room with acute gastrointestinal bleeding (hereafter referred to as AGIB), and assess whether physicians' decisions to either hospitalize or discharge each patient appear to be capturing information which is not available to the Glasgow-Blatchford Score (GBS). The GBS is a standardized measure of risk which is known to be a highly sensitive indicator for whether a patient presenting with AGIB will indeed require hospitalization (findings which we corroborate below). However, despite the excellent performance of this algorithmic risk score, we might be understandably hesitant to automate triage decisions without any physician oversight. As just one example, anticoagulant medications ('blood thinners') are known to exacerbate the risk of severe bleeding. However, whether or not a patient is taking anticoagulant medication is not included as a feature in the construction of the Glasgow-Blatchford score, and indeed may not even be recorded in the patient's electronic health record (if, for example, they are a member of an underserved population and have had limited prior contact with the healthcare system). This is one of many additional factors an emergency room physician might elicit directly from the patient to inform an admit/discharge decision. We thus seek to answer the following question:
Do emergency room physicians usefully incorporate information which is not available to the Glasgow-Blatchford score?
We answer in the affirmative, demonstrating that although the GBS provides risk scores which are highly competitive with (and indeed, arguably better than) physicians' discretionary decisions, there is strong evidence that physicians are incorporating additional information which is not captured in the construction of the GBS. Before presenting our results, we first provide additional background about this setting.
Background: risk stratification and triage for gastrointestinal bleeding. Acute gastrointestinal bleeding is a potentially serious condition for which 530,855 patients/year receive treatment in the United States alone (Peery et al. (2022)). It is estimated that  $32\%$  of patients with presumed bleeding from the lower gastrointestinal tract (Oakland et al. (2017)) and  $45\%$  of patients with presumed bleeding from the upper gastrointestinal tract (Stanley et al. (2017)) require urgent medical intervention; overall mortality rates for AGIB in the U.S. are estimated at around 3 per 100,000 (Peery et al. (2022)). For patients who present with AGIB in the emergency room, the attending physician is tasked with deciding whether the bleeding is severe enough to warrant admission to the hospital. However, the specific etiology of AGIB is often difficult to determine from patient presentation alone, and gold standard diagnostic techniques – an endoscopy for upper GI bleeding or a colonoscopy for lower GI bleeding – are both invasive and costly, particularly when performed urgently in a hospital setting. To aid emergency room physicians in making this determination more efficiently, the Glasgow-Blatchford Bleeding Score or GBS (Blatchford et al. (2000)) is a standard screening metric used to assess the risk that a patient with acute upper GI bleeding will require red blood cell transfusion, intervention to stop bleeding, or die within 30 days. It has been also validated in patients with acute lower gastrointestinal bleeding<sup>3</sup> to assess need for intervention to stop bleeding or risk of death (Asad Ur-Rahman and Abusaada (2018)); accordingly, we interpret the GBS as a measure of risk for patients who present with either upper or lower GI bleeding in the emergency department.
Construction of the Glasgow-Blatchford Score. The Glasgow-Blatchford Score is a function of the following nine patient characteristics: blood urea nitrogen (BUN), hemoglobin (HGB), systolic blood pressure (SBP), pulse, cardiac failure, hepatic disease, melena, syncope and biological sex. The first four are real-valued and the latter five are binary. The GBS is calculated by first converting the continuous features to ordinal values (BUN and HGB to 6 point scales, SBP to a 3 point scale and pulse to a binary value) and then summing the values of the first 8 features. Biological sex is used to inform the conversion of HGB to an ordinal value. Scores are integers ranging from 0 to 23,
$^{3}$ US and international guidelines use the Glasgow-Blatchford score as the preferred risk score for assessing patients with upper gastrointestinal bleeding (Laine et al. (2021); Barkun et al. (2019)). Other risk scores tailored to bleeding in the lower gastrointestinal tract have been proposed in the literature, but these are less commonly used in practice. We refer interested readers to Almaghrabi et al. (2022) for additional details.
with higher scores indicating a higher risk that a patient will require subsequent intervention. US and international guidelines suggest that patients with a score of 0 or 1 can be safely discharged from the emergency department (Laine et al. (2021); Barkun et al. (2019)), with further investigation to be performed outside the hospital. For additional details on the construction of the GBS, we refer to Blatchford et al. (2000).
Defining features, predictions and outcomes. We consider a sample of 3617 patients who presented with AGIB at one of three hospitals in a large academic health system between 2014 and 2018. Consistent with the goals of triage for patients with AGIB, we record an 'adverse outcome' if a patient (1) requires some form of urgent intervention to stop bleeding (endoscopic, interventional radiologic, or surgical; excluding patients who only undergo a diagnostic endoscopy or colonoscopy) while in the hospital (2) dies within 30 days of their emergency room visit or (3) is initially discharged but later readmitted within 30 days. $^{4}$  As is typical of large urban hospitals in the United States, staffing protocols at this health system dictate a separation of responsibilities between emergency room physicians and other specialists. In particular, while emergency room physicians make an initial decision whether to hospitalize a patient, it is typically a gastrointestinal specialist who subsequently decides whether a patient admitted with AGIB requires some form of urgent hemostatic intervention. This feature, along with our ability to observe whether discharged patients are subsequently readmitted, substantially mitigates the selective labels issue that might otherwise occur in this setting. Thus, consistent with clinical and regulatory guidelines – to avoid hospitalizing patients who do not require urgent intervention (Stanley et al. (2009)), and to avoid discharging patients who are likely to be readmitted within 30 days (NEJM (2018)) – we interpret the emergency room physician's decision to admit or discharge a patient as a prediction that one of these adverse outcomes will occur. We thus instantiate our model by letting  $X_{i} \in [0,1]^{9}$  be the nine discrete patient characteristics from which the Glasgow-Blatchford Score is computed $^{5}$ ; the only transformation we apply is to normalize each feature to lie in  $[0,1]$ . We further let  $\hat{Y}_{i} \in \{0,1\}$  indicate whether that patient was initially hospitalized, and  $Y_{i} \in \{0,1\}$  indicate whether that patient suffered one of the adverse outcomes defined above.
Assessing the accuracy of physician decisions. We first summarize the performance of the emergency room physicians' hospitalization decisions, and compare them with the performance of a simple rule which would instead admit every patient with a GBS above a certain threshold and discharge the remainder (Table 1). We consider thresholds of 0 and 1 – the generally accepted range for low risk patients (Laine et al. (2021); Barkun et al. (2019)) – as well as less conservative thresholds of 2 and 7 (the latter of which we find maximizes overall accuracy). For additional context, we also provide the total fraction of patients admitted under each decision rule.
Table 1: Comparing the accuracy of physician hospitalization decisions ('Physician Discretion') to those made by thresholding the GBS. For example, 'Admit  $\mathrm{GBS} > 1$  'hospitalizes patients with a GBS strictly larger than 1. 'Fraction Hospitalized' indicates the fraction of patients hospitalized by each rule. 'Accuracy' indicates the  $0 / 1$  accuracy of each rule, where a decision is correct if it hospitalizes a patient who suffers an adverse outcome (as defined above) or discharges a patient who does not. 'Sensitivity' indicates the fraction of patients who suffer an adverse outcome that are correctly hospitalized, and 'Specificity' indicates the fraction of patients who do not suffer an adverse outcome that are correctly discharged. Results are reported to  $\pm 2$  standard errors.
<sup>4</sup>This threshold is consistent with the definition used in the Centers for Medicare and Medicaid Services Hospital Readmission Reduction Program, which seeks to incentivize healthcare providers to avoid discharging patients who will be readmitted within 30 days (NEJM (2018))
5We consider alternative definitions of the feature space in Appendix E, where we find substantively similar results.
Unsurprisingly, we find that the physicians calibrate their decisions to maximize sensitivity (minimize false negatives) at the expense of admitting a significant fraction of patients who, in retrospect, could have been discharged immediately. Indeed we find that although  $86\%$  of patients are initially hospitalized, only  $\approx 42\%$  actually require a specific hospital-based intervention or otherwise suffer an adverse outcome which would justify hospitalization (i.e.,  $y_{i} = 1$ ). Consistent with Blatchford et al. (2000) and Chatten et al. (2018), we also find that thresholding the GBS in the range of  $[0, 2]$  achieves a sensitivity of close to  $100\%$ . We further can see that using one of these thresholds may achieve overall accuracy (driven by improved specificity) which is substantially better than physician discretion. Nonetheless, we seek to test whether physicians demonstrate evidence of expertise in distinguishing patients with identical (or nearly identical) scores.
Testing for physician expertise. We now present the results of running ExpertTest for  $K = 1000$  resampled datasets and various values of  $L$  (where  $L = 1808$  is the largest possible choice given  $n = 3617$ ) in Table 2. We define the distance metric  $m(x_{1},x_{2})\coloneqq ||x_{1} - x_{2}||_{2}$ , though this choice is inconsequential when the number of 'mismatched pairs' (those pairs  $x,x^{\prime}$  where  $x\neq x^{\prime}$ ) is 0.
We also observe that, in the special case of binary predictions and outcomes, it is possible to analytically determine the number of swaps which increase or decrease the value of nearly any natural loss function  $F(\cdot)$ . Thus, although we let  $F(D) \coloneqq \frac{1}{n} \sum_{i} \mathbb{1}[y_i \neq \hat{y}_i]$  for concreteness, our results are largely insensitive to this choice; in particular, they remain the same when false negatives and false positives might incur arbitrarily different costs. We elaborate on this phenomenon in Appendix E.
Table 2: The results of running ExpertTest, where each pair of patients is chosen to be as similar as possible with respect to the nine (discrete) inputs to the Glasgow-Blatchford score.  $L$  indicates the number of pairs selected for the test, of which 'mismatched pairs' are not identical to each other. Swaps that decrease (respectively, increase) loss indicates how many of the  $L$  pairs result in a decrease (respectively, increase) in the 0/1 loss when their corresponding hospitalization decisions are exchanged with each other.  $\tau$  is the p-value obtained from running ExpertTest.
As the results demonstrate, there is very strong evidence that emergency room physicians incorporate information which is not available to the Glasgow-Blatchford score. In particular, our test indicates that physicians can reliably distinguish patients who appear identical with respect to the nine features considered by the GBS – and can make hospitalization decisions accordingly – even though simple GBS thresholding is highly competitive with physician performance. This implies that no predictive algorithm trained on these nine features, even one which is substantially more complicated than the GBS, can fully capture the information that physicians use to make hospitalization decisions.
To interpret the value of  $\tau$ , observe that for  $L \geq 500$  we recover the smallest possible value  $\tau = 1 / (K + 1) = 1 / 1001$ . Furthermore, for all but the final experiment, the number of mismatched pairs is 0, which means there is no additional type one error incurred (i.e.,  $\varepsilon_{n,L}^{*}$  (11) is guaranteed to be 0). For additional intuition on the behavior of ExpertTest, we refer the reader to the synthetic experiments in Appendix F.
6 Discussion and limitations
In this work we provide a simple test to detect whether a human forecaster is incorporating unobserved information into their predictions, and illustrate its utility in a case study of hospitalization decisions made by emergency room physicians. A key insight is to recognize that this requires more care than simply testing whether the forecaster outperforms an algorithm trained on observable data; indeed, a large body of prior work suggests that this is rarely the case. Nonetheless, there are many settings in which we might expect that an expert is using information or intuition which is difficult to replicate with a predictive model.
An important limitation of our approach is that we do not consider the possibility that expert forecasts might inform decisions which causally effect the outcome of interest, as is often the case in practice. We also do not address the possibility that the objective of interest is not merely accuracy, but perhaps some more sophisticated measure of utility (e.g., one which also values fairness or simplicity); this is explored in Rambachan (2022). We caution more generally that there are often normative reasons to prefer human decision makers, and our test captures merely one possible notion of expertise. The results of our test should thus not be taken as recommending the automation of a given forecasting task.
Furthermore, while our framework is quite general, it is worth emphasizing that the specific algorithm we propose is only one possible test of  $H_0$ . Our algorithm does not scale naturally to settings where  $\mathcal{X}$  is high-dimensional, and in such cases it is likely that a more sophisticated test of conditional independence (e.g. a kernel-based method; see Fukumizu et al. (2004), Gretton et al. (2007) and Zhang et al. (2011), among others) would have more power to reject  $H_0$ . Another possible heuristic is to simply choose some learning algorithm to estimate (e.g.)  $\mathbb{E}[Y \mid X]$  and  $\mathbb{E}[Y \mid X, \hat{Y}]$ , and examine which of the two provides better out of sample performance. This can be viewed as a form of feature selection; indeed the 'knockoffs' approach of Candès et al. (2016) which inspires our work is often used as a feature selection procedure in machine learning pipelines. However, most learning algorithms do not provide p-values with the same natural interpretation we describe in section 3, and we thus view these approaches as complementary to our own.
Finally, our work draws a clean separation between the 'upstream' inferential goal of detecting whether a forecaster is incorporating unobserved information and the 'downstream' algorithmic task of designing tools which complement or otherwise incorporate human expertise. These problems share a very similar underlying structure however, and we conjecture that - as has been observed in other supervised learning settings, e.g. Kearns et al. (2018) - there is a tight connection between these auditing and learning problems. We leave an exploration of these questions for future work.
Agrawal (2019). The economics of artificial intelligence. (NBER) National Bureau of Economic Research Conference Reports. University of Chicago Press, Chicago, IL.
Agrawal, A., Gans, J., and Goldfarb, A. (2018). Exploring the impact of artificial intelligence: Prediction versus judgment. Technical report, National Bureau of Economic Research, Cambridge, MA.
Almaghrabi, M., Gandhi, M., Guizzetti, L., Iansavichene, A., Yan, B., Wilson, A., Oakland, K., Jairath, V., and Sey, M. (2022). Comparison of risk scores for lower gastrointestinal bleeding: A systematic review and meta-analysis. *JAMA Netw. Open*, 5(5):e2214253.
Arnold, D., Dobbie, W., and Hull, P. (2020). Measuring racial discrimination in bail decisions. Technical report, National Bureau of Economic Research, Cambridge, MA.
Asad Ur-Rahman, Jian Guan, S. K. A. M. M. S. E. I. X. H. A. M. and Abusaada, K. (2018). Both full glasgow-blatchford score and modified glasgow-blatchford score predict the need for intervention and mortality in patients with acute lower gastrointestinal bleeding. *Digestive Diseases and Sciences*, 63:3020-3025.
Bansal, G., Wu, T. S., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., and Weld, D. S. (2020). Does the whole exceed its parts? the effect of ai explanations on complementary team performance. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems.
Barber, R. F. and Candès, E. (2019). On the construction of knockoffs in case-control studies. Stat, 8(1):e225. e225 sta4.225.
Barber, R. F., Candès, E. J., and Samworth, R. J. (2018). Robust inference with knockoffs.
Barkun, A. N., Almadi, M., Kuipers, E. J., Laine, L., Sung, J., Tse, F., Leontiadis, G. I., Abraham, N. S., Calvet, X., Francis K.L. Chan, J. D., Enns, R., Gralnek, I. M., Jairath, V., Jensen, D., Lau, J., Lip, G. Y., Loffroy, R., Maluf-Filho, F., Meltzer, A. C., Reddy, N., Saltzman, J. R., Marshall, J. K., and Bardou, M. (2019). Management of nonvariceal upper gastrointestinal
Stanley, A. J., Ashley, D., Dalton, H. R., Mowat, C., Gaya, D. R., Thompson, E., Warshow, U., Groome, M., Cahill, A., Benson, G., Blatchford, O., and Murray, W. (2009). Outpatient management of patients with low-risk upper-gastrointestinal haemorrhage: multicentre validation and prospective evaluation. Lancet, 373(9657):42-47.
Stanley, A. J., Laine, L., Dalton, H. R., Ngu, J. H., Schultz, M., Abazi, R., Zakko, L., Thornton, S., Wilkinson, K., Khor, C. J. L., Murray, I. A., and Laursen, S. B. (2017). Comparison of risk scoring systems for patients presenting with upper gastrointestinal bleeding: international multicentre prospective study. BMJ, 356.
Tversky, A. and Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185(4157):1124-1131.
Zhang, K., Peters, J., Janzing, D., and Scholköpf, B. (2011). Kernel-based conditional independence test and application in causal discovery. In Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence, UAI'11, page 804-813, Arlington, Virginia, USA. AUAI Press.
Human decision making, interplay with algorithms. Our work contributes to a vast literature on understanding how humans, and particularly human experts, make decisions. We do not attempt to provide a comprehensive summary of this work, but refer the reader to Tversky and Kahneman (1974) and Camerer and Johnson (1991) for general background. Of particular relevance for our setting is work which investigates whether humans make systematic mistakes in their decisions, which has been studied in the context of bail decisions (Kleinberg et al. (2017), Rambachan (2022), Lakkaraju et al. (2017) and Arnold et al. (2020)), college admissions (Kuncel et al. (2013), Dawes (1971)) and patient triage and diagnosis (Currie and MacLeod (2017), Mullainathan and Obermeyer (2019)) among others. One common theme in these works is that the decision made by the human expert will often influence the outcome of interest; for example, an emergency room doctor's initial diagnosis will inform the treatment a patient receives, which subsequently affects their health outcomes. Furthermore, it is often the case that even observing the outcome of interest is contingent on the human's decision: for example, in a college admissions setting, we might only observe historical outcomes for admitted students, which makes it challenging to draw inferences about applicants. This one-sided labeling problem is a form of endogeneity which has been well studied in the context of causal inference, and these works often adopt a causal perspective to address these challenges.