Auditing for Human Expertise
Authors: Rohan Alur
Loren Laine
Darrick K Li
Manish Raghavan
Devavrat Shah
Dennis Shung
Venue: NeurIPS 2023 spotlight (2023)
Keywords: hypothesis testing
human-AI complementarity
machine learning for healthcare
TL;DR: We develop a statistical framework to assess whether human experts add value which could not be captured by any learning algorithm for a given prediction task.

Abstract: High-stakes prediction tasks (e.g., patient diagnosis) are often handled by trained human experts. A common source of concern about automation in these settings is that experts may exercise intuition that is difficult to model and/or have access to information (e.g., conversations with a patient) that is simply unavailable to a would-be algorithm. This raises a natural question whether human experts add value which could not be captured by an algorithmic predictor.
We develop a statistical framework under which we can pose this question as a natural hypothesis test. Indeed, as our framework highlights, detecting human expertise is more subtle than simply comparing the accuracy of expert predictions to those made by a particular learning algorithm. Instead, we propose a simple procedure which tests whether expert predictions are statistically independent from the outcomes of interest after conditioning on the available inputs (‘features’). A rejection of our test thus suggests that human experts may add value to any algorithm trained on the available data, and has direct implications for whether human-AI ‘complementarity’ is achievable in a given prediction task.
We highlight the utility of our procedure using admissions data collected from the emergency department of a large academic hospital system, where we show that physicians’ admit/discharge decisions for patients with acute gastrointestinal bleeding (AGIB) appear to be incorporating information that is not available to a standard algorithmic screening tool. This is despite the fact that the screening tool is arguably more accurate than physicians’ discretionary decisions, highlighting that – even absent normative concerns about accountability or interpretability – accuracy is insufficient to justify algorithmic automation.

Auditing for Human Expertise
Rohan Alur
Loren Laine
Darrick K. Li School of Medicine Yale darrick. 1 li@lllllll.... edu
EECS MIT ralurêmite edu edu
School of Medicine Yale loren. www.aaaaale.e.eee. edu
Manish Raghavan Sloan, EECS MIT mraghCmit.edi
Devavrat Shah EECS, IDSS, LIDS, SDSC MIT devavrat
Dennis Shung School of Medicine
Yale
dennis. shungdyyu edu
Abstract
High-stakes prediction tasks (e.g., patient diagnosis) are often handled by trained human experts. A common source of concern about automation in these settings is that experts may exercise intuition that is difficult to model and/or have access information (e.g., conversations with patient) that simply unavailable a would-be algorithm. This raises natural question whether human experts add value which could not be captured by an algorithmic predictor.
We develop a statistical framework under which we can pose this question as a natural hypothesis test. Indeed, expertise more subtle than simply comparing the accuracy of expert predictions those made by particular learning algorithm. Instead, propose simple procedure which tests whether expert predictions are statistically independent from the outcomes of interest after conditioning on the available inputs ('features') ). A rejection of our test thus suggests that human experts may add value to any algorithm trained on the available data, and has direct implications for whether human-AI 'complementarity is achievable in a given prediction task.
framework highlights, detecting human
We highlight the utility of our procedure using admissions data collected from the emergency department of a large academic hospital system, where we show that physicians' admit/dischagge decisions for patients with acute gastrointestinal bleeding (AGIB) appear to be incorporating information that is not available to a standard algorithmic screening tool. This is despite the fact that the screening tool is arguably more accurate than physicians' discretionary decisions, highlighting that even absent normative concerns about accountability or interpretability accuracy is insufficient to justify algorithmic automation.
Introduction
Progress in machine learning, and in algorithmic decision aids more generally, has raised the prospect that algorithms may complement or even automate human decision making in wide variety of settings. implemented carefully, these tools have the potential to improve accuracy, fairness, interpretability and consistency in many prediction and decision tasks. However, primary challenge in nearly all such settings that some the relevant inputs 'features," in machine learning parlance difficult even impossible to encode in way that an algorithm easily consume. For example, doctors use direct conversations with patients to inform their diagnoses, and sports franchises employ professional scouts to qualitatively assess prospective players. One can think of these experts as incorporating information which is practically difficult to provide to an algorithm, particularly as
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

tabular data, or perhaps exercising judgment which is infeasible to replicate with a computational process. Either perspective presents a challenge when deploying predictive algorithmic tools, as any such model will necessarily fail to incorporate at least some of the information that a human expert might consider. Thus, as we seek to use algorithms to improve decision-making we must answer the following question:
For a given prediction task, do human experts add value which could not be captured by any algorithmic forecasting rule?
The answer to this question has significant consequences: if experts are incorporating salient but hard-to-quantify information, we might attempt to somehow ensemble or combine the human and algorithmic predictions; this is commonly referred as seeking 'complementarity in the literature on human- manhine interaction. On the other hand, if it appears that an expert is not extracting signal beyond whatever is contained in the available features, we might consider whether we can automate the prediction task entirely, or at least reduce the degree to which a human may override algorithmic recommendations.
At this stage it is worth asking - why not simply compare the prediction accuracy of human expert to that of a particular predictive algorithm? If the human expert performs better than a competing algorithm, we might say the expert adds value which is not captured by the algorithm. However, as the example presented next illustrates, it is possible for the expert to incorporate information that could not be captured by any learning algorithm even when the expert substantially underperforms a particular algorithm trained accomplish the same task. Indeed, this not just hypothetical: a large body of prior literature (see e.g. Agrawal (2019) for comprehensive overview) suggests that humans reliably underperform even simple statistical models, and Section 5, we find exactly this dynamic in real-world patient triage data. Nonetheless, as we highlight next, humans may still add valuable information in a given forecasting task.
An illustration: experts may add information despite poor predictions. Let Y denote the outcome of interest and let X, U be features that drive the outcome. Specifically, let
Y = X U + E1:
where 61 is some exogenous noise. For the purposes of this stylized example, we'll assume that X, U,U,6 61 are all zero mean and pairwise independent random variables. Suppose the human expert can observe both X and U, but only X is made available to a predictive algorithm. An algorithm tasked with minimizing squared error might then seek precisely estimate [[[ X. In contrast, the expert may instead use simpler heuristics to construct an estimate which can be modeled as
(X) sign(U) E2,
(2)
where is independent zero-mean noise, and can be thought of as modeling idiosyncracies in the expert's cognitive process'. As discussed in detail in Appendix there exist natural distributions over (X, U, 61, E2) such that the algorithm performs substantially better than the expert in terms of predictive accuracy. In fact, we show that there exist natural distributions where the algorithm outper- forms the expert even under any linear post-processing of Y (e.g.. to correct for expert predictions which are highly correlated with Y but perhaps incorrectly centered or scaled). Nonetheless, the expert predictions clearly contain information (ef. sign(U)) that is not captured by the algorithm.
However, because U is not recorded in the available data, it is not obvious how to distinguish the above scenario from one in which the expert only extracts signal from X. For example, they might instead make predictions as follows:
sign 62.
(3)
While a learning algorithm may outperform the expert in both cases, the expert in scenario (2) still captures valuable information; the expert in (3) clearly does not. The goal of this work will be to develop a test which allows us to distinguish between scenarios like these without the strong modeling assumptions made in this example.
For example, well-known study by Eren and Mocan (2018) demonstrates that unexpected losses by the Louisiana State University football team lead judges to hand out longer juvenile sentences; this is a form of capricious decision making which will manifest noise (e2) in analysis of sentencing decisions.

---

Contributions. To understand whether human experts can add value for given prediction task, we develop a statistical framework under which answering this question becomes a natural hypothesis test. We then provide simple, data-driven procedure to test this hypothesis. Our proposed algorithm takes the form of a conditional independence test, and is inspired by the Model-X Knockoffs framework of Candès et al. (2016), the Conditional Permutation Test of Berrett et al. (2018) and the 'Model- Powered' test of Sen et al. (2017). Our test is straightforward to implement and provides transparent, interpretable p-values.
Our work is closely related to a large body of literature comparing human performance to that of an algorithm (Cowgill (2018), Dawes et al. (1989), Grove et al. (2000), among others), and developing learning algorithms which are complementary to human expertise (Madras et al. (2018). Raghu et al. (2019), Mozannar and Sontag (2020), Keswani et al. (2021), Agrawal et al. (2018), Bansal et al. (2020), Rastogi al. (2022) and Bastani et al. (2021)). However, although similarly motivated, we address a different problem which is in sense 'upstream' of these works, as we are interested in testing for whether a human forecaster demonstrates expertise which cannot be replicated by any algorithm. Thus, we think of our test as assessing complementarity; success in practice will further depend on the ability of a mechanism designer to actually incorporate human expertise into some particular algorithmic pipeline or feedback system. We discuss these connections further in Appendix A.
necessary condition for achieving human-AI
We apply our test to evaluate whether emergency room physicians incorporate valuable information which is not available to a common algorithmic risk score for patients with acute gastrointestinal bleeding (AGIB). To that end, we utilize patient admissions data collected from the emergency department of large academic hospital system. Consistent with prior literature, we find that this algorithmic score is an exceptionally sensitive measure of patient risk and one that is highly competitive with physicians expert assessments. Nonetheless, our test provides strong evidence that physician decisions to either hospitalize or discharge patients with AGIB are incorporating valuable information that is not captured by the screening tool. Our results highlight that prediction accuracy is not sufficient to justify automation of a given prediction task. Instead, our results make a case for experts working with a predictive algorithm, even when algorithms might handily outperform their human counterparts.
Organization. In Section 2, we formalize the problem of auditing for expertise, and in Section 3 we present our data-driven hypothesis test. Section 4 then examines the theoretical properties of the test. In Section 5 we present our empirical findings from applying this test real-world patient triage data. Finally, Section 6 provides discussion of our results and directions for future work. We also include discussion of additional related work in Appendix A, and provide numerical simulations to corroborate our thcoretical and empirical results in Appendix F?.
2 Setup and question of interest
We consider generic prediction task in which the goal is to forecast some outcome Y R on the basis of observable features X e X. The human expert may additionally have access to some auxiliary private information U E U. For concreteness, let X = Rd for some d > 1.
We posit that the outcome Y is generated as follows: for some unknown function f X x 1 TR,
Y f(X, U) +
where, without loss of generality, e represents mean zero idiosyncratic noise with unknown variance.
We are also given predictions by human expert, denoted as Y. We posit that the expert predictions Y are generated as follows: for some unknown function ff X U + R.
Ý ((X,U)+es
(5)
where Eo also captures mean zero idiosyncratic noise with unknown variance.
We observe (X, Y, Y) which obey (4)-(5); the private auxiliary feature U is not observed. Concretely, we observe n data points (I;, yi, 9z), i E [r]] = n...
*Code, data and instructions to replicate our experiments are available at https//ggttub..ommaaaaauddddd com/ralur/auddting human- expertise. Publication of the results and data associated with the empirical study in section 5 have been approved by the relevant institutional review board (IRB).
3

---

Our goal is to answer the question "do human experts add information which could not be captured by any algorithm for a given prediction task?" We assume that any competing learning algorithm can utilize X to predict Y, but it can not utilize U. Thus, our problem reduces to testing whether U has some effect both Y and Y. which corresponds to the ability of an expert to extract signal about Y from U. If instead U has no effect on Y, Y or both (either because U is uninformative about Y or the expert is unable to perceive this efffec), then conditioned on X, Y and Y are independent. That is, if the human expert fails to add any information which could not be extracted from the observable features X, the following must hold:
Ho : Y 11 8 X.
(6)
Intuitively, Ho captures the fact that once we observe X, Y provides no additional information about Y unless the expert also making use of some unobserved signal U (whether explicitly or implicitly). In contrast, the rejection of Ho should be taken as an evidence that the expert (or experts) can add value to any learning algorithm trained on the observable features X E X; indeed, a strength of this framework is that it does not require specifying particular algorithmic baseline. However, it's worth remarking that an important special case is take to be the prediction made by some specific learning algorithm trained to forecast Y. In this setting, our test then reduces to assessing whether Y adds information to the predictions made by this learning algorithm, and can be viewed form of feature selection.
Goal. Test the null hypothesis Ho using observed data (æi, Di, 0:), i E [x]] = (1....,n))
To make this model concrete, in Section 5 we use framework test whether emergency room physicians incorporate information that is not available to a common algorithmic risk score when deciding whether to hospitalize patients. Accordingly, we let X [0, be the inputs to the risk score, Y {0, 1} be binary variable indicating whether a given patient was hospitalized, and Y E 10, } be an indicator for whether, in retrospect, a patient should have been hospitalized. The risk score alone turns out to be a highly accurate predictor of Y. but physicians take many other factors into account when making hospitalization decisions. We thus seek to test whether physicians indeed extract signal which is not available to the risk score (Y A Y X), or whether attempts to incorporate other information and/or exercise expert judgement simply manifest as noise (Y Y X).
ExpertTest: a statistical test for human expertise
To derive statistical test of Ho. we will make use of the following elementary but powerful fact about exchangeable random variables.
A test for exchangeability. Consider K + random variables (Zo.... Zx) which are exchangeable, i.e. the joint distribution of tation a: : {O...., K} + ....... K). KK). For example, if .... Zx are independent and identically distributed (i..d.), then they are exchangeable. Let J' be a function that maps these variables to a real value. For any such F--). can be verified that the order statistics (with any ties broken uniformly at random) of F(Zo)..... F(Zx) are uniformly distributed over (K + 1)! permutations of (O.... K". That is, Tk defined next, is distributed uniformly over (0, 1/ K,K/K,K..., 1]:
..., , ( for any permu- Zx) ) identical to that of (Zo(o):
K (F F(Z))
TK
K
where we use definition 1/a S = 1f < B and 0 if > B. If instead a = 3, we independently assign it to be 1 or 0 with equal probability. Thus, if (Zo.... Zx) are exchangeable, then P(TK S
a) S a + 1/(K 1) K=00 and with p-value (effectively) equal to Tk.
reject the hypothesis that (Zo...., Zx) exchangeable
Observe that while this validity guarantee holds for any choice of F(-), the power of the test will depend crucially on this choice; for example, a constant function which maps every argument to the same value would have power to reject the null hypothesis. We return to the choice of F()) below.

---

close as Constructing exchangeable distributions. We will leverage the prior fact about the order statistics
of exchangeable random variables to design test of Ho Y L y X. In particular, we would like to use the observed data to construct K + 1 random variables that are exchangeable under Ho. but not exchangeable otherwise. To that end, consider a simplified setting where = 2, with 31 = 22 = x. Thus, our observations are Zo = {ar, 31, 91 (æt y2, (j2)]. Suppose we now sample (W1, 122) uniformly at random from {(11 122, (1/,, 1/1 ). That is, we swap the observed values (1/1 92)
with probability to construct a new dataset Z1 = {x, y1. B1), (z, V2: F2)}
Under Ho. it is straightforward to show that Zo and 21 are independent and identically distributed con- ditioned on observing (z), z), (y), M22 and either (#1. (2) or (52. 21). That is. Zo. Zi are exchangeable, which will allow us to utilize the test described above for Ho.
Why condition this somewhat complicated event? Intuitively, (ū1, 122 from the distribution of D x under the null, (2r, y, 9) and (at, y, 1/) will be exchangeable by definition. However, this requires that we know (or can accurately estimate) the distribution of Y X, which in turn requires modeling the expert's decision making directly. Instead, we simplify the resampling process by only considering a swap of the observed ý values between identical values of x - this guarantees exchangeable data without modeling Dyix x at all!
would like resample y
This approach can be extended for n larger than 2. Specifically, if there are L pairs of identical values, i.c. T2--1 T2e for 1 < l < L. then possible construct i..d. Zo...., .,, Zx for larger K by randomly exchanging values of for each pair of data points.
As discussed above, we'll also need choose particular function F(-) to apply Zo and Zi. A natural, discriminatory choice of is a loss function: for example, given D = [ Vi. ú:) : S 2 let F(D)= E, [y: - ú.)². This endows TK with a natural interpretation it is the probability that an expert could have performed as well as they did (with respect to the chosen loss function F) by pure chance, without systematically leveraging some unobserved U.
Of course, in practice we are unlikely to observe many pairs where xx = Z2e. particularly when x takes value in a non-finite domain, e.g. [0, 1] or R. However, if the conditional distribution of y|x is nearly the same for close enough values of X and X x', then we can use similar approach with some additional approximation error. This is precisely the test that we describe next.
ExpertTest. Let L 2 be an algorithmic parameter and X' x X' + Rzo be some distance metric over X, e.g. the & distance. Let F(-) be some loss function of interest, e.g. the mean squared error.
First, compute m( xz)) : + [n] and greedily select L disjoint pairs which possible under m(.-.). Denote these pairs by (( Di,,) : & [L]}.
(i2--1, i2 l [T]}} denote the observed dataset restricted to the I. chosen pairs. Let Dj be an additional dataset generated by independently swapping each pair (0ize-1> 1. Dige) ) with probability 1 1/2, and repeat this resampling procedure to generate D.... Dg. Next,
Let Do {(1;, Vr. Ji)
compute TK as follows:
TK K
S(()))
(8)
Finally, reject the hypothesis Ho with p-value a a++1 1/(K (K++) ([)K+))iff if for any desired confidence level a E (0, 1). Our test is thus quite simple: find L pairs of points that are close under some distance metric m(..)). and create K synthetic datasets by swapping the expert forecasts for each pair independently with probability 1/2. the expert's loss on the original dataset is "small" relative to the loss these resampled datasets, this is evidence that the synthetic datasets not exchangeable with the original, and thus, the expert is using some private information U.
Of course, unlike in the example above, we swapped pairs of predictions for different values of xr Thus, Do... Dr are not exchangeable under Hg. However, we'll argue that because we paired "nearby" values of z, these datasets are "nearly" exchangeable. These are the results we present next.
5

---

4 Results
We provide theoretical guarantees associated with the ExpertTest. First, we demonstrate the validity of our test in a generic setting. That is, if Ho is true, then ExpertTest will not reject it with high probability. We then quantify this guarantee precisely under a meaningful generative model.
To state the validity result, we need some notation. For any (ar, () and (x', f'), define the odds ratio
Q(Ý X Q( = 1' X
r((2; 0). (d'.
Q(Y = 5' X = x) x Q( = v X = x')
where Q(-1-) represents the density of the conditional distribution of human predictions &IX under Ho- For simplicity, we assume that such a conditional density exists.
Theorem (Validity of ExpertTest) Given a E (0, 1) and parameters K 2 1, 2 1, the Type I error of ExpertTest satisfies
P(TK (a) aa (1- )4) + K+1
(10)
Where =,, is defined as follows
E max EELL
(1I)
r(( Mize-1) (Tize: Di2e)
2
We remark briefly on the role of the parameters L and K in this result. To begin with, K+I embedded in the type I error, and thus taking the number of resampled datasets K to be as large as possible (subject only to computational constraints) sharpens the validity guarantee. We also observe that the bound becomes weaker as increases. However, observe also that ExpertTest is implicitly using an L-dimensional distribution (or L fresh samples) to reject Ho, which means that increasing also provides additional power to the test.
Notice also that the odds ratio (9) is guaranteed to be 1 if x = x', regardless of the underlying distribution D. This a coincidence, and our test is based implicitly on the heuristic that the odds ratio will tend away from 1 as the distance m(x, æ²) increases (we quantify this intuition precisely below). Thus, increasing L will typically also increase = because larger values of will force us to pair additional observations (ar, x') which are farther apart under the distance metric.
The type one error bound (10) suggests that we can balance the trade off between validity and power when _ L < or o(1), the right hand side of (10) reduces to a + o(1). Next we describe representative generative setup where there is natural choice of that leads e eh = o(1).
Generative model. Let X' [0, 1j c Rd. Let the conditional density of the human expert's forecasts Q(-[x)) be smooth. Specifically, for any x, x E =0,11
Qū X = r) Q X z') < <1+Cx 1
sup GER
(12)
for some constant C > O. Under this setup, Theorem reduces to the following.
Theorem (Asymptotic Validity) Given a 10 and under (12), with the appropriate choice of L I, the type I error of ExpertTest satisfies
P(TK S a) aalo
(13)
as n. K + 00.
Intuitively, (12) is intended to model a forecasting rule which is 'simple,' in the sense that human experts don't finely distinguish between instances whose feature vectors are close under the & norm. Importantly, this does not rule out the possibility that predictions for two specific (x,,²) instances could differ substantially only that the distributions Y X and Ý X x' are similar when x 2 s'.
We make no such assumption about Dyjx. the conditional distribution of the true outcomes.
Proofs of theorems and can be found in Appendices B and C respectively. We now illustrate the utility of our test with an empirical study of physician hospitalization decisions.

---

5 A case study: physician expertise in emergency room triage
Emergency room triage decisions present a natural real-world setting for our work, as we can assess whether physicians make hospitalization decisions by incorporating information which is not available to an algorithmic risk score. We consider the particular case of patients who present in the emergency room with acute gastrointestinal bleeding (hereafter referred to as AGIB), and assess whether physicians' decisions either hospitalize or discharge each patient appear to be capturing information which is not available to the Hlasgoww-latchford Score (GBS). The GBS standardized measure of risk which is known to be a highly sensitive indicator for whether a patient presenting with AGIB will indeed require hospitalization (findings which we corroborate below). However, despite the excellent performance of this algorithmic risk score, we might be understandably hesitant to automate triage decisions without any physician oversight. As just one example, anticoagulant medications ('blood thinners') are known to exacerbate the risk of severe bleeding. However, whether or not a patient is taking anticoagulant medication is not included as feature in the construction of the Glasgow- Glasgow-Blatchford score, and indeed may not even be recorded in the patient's 's electronic health record (if, for example, they are member of underserved population and have had limited prior contact with the healthcare system). This is one of many additional factors an emergency room physician might elicit directly from the patient inform an admit/discharge decision. We thus seek to answer the following question:
Do emergency room physicians usefully incorporate information which is not available to the Blatcch-o score?
We answer in the affirmative, demonstrating that although the GBS provides risk scores which are highly competitive with (and indeed, arguably better than) physicians' discretionary decisions, there is strong evidence that physicians are incorporating additional information which is not captured in the construction of the GBS. Before presenting our results, we first provide additional background about this setting.
Background: risk stratification and triage for gastrointestinal bleeding. Acute gastrointestinal bleeding is a potentially serious condition for which 530,855 patients/year receive treatment in estimated that 32% of patients with presumed
the United States alone (Peery et al. (2022)). bleeding from the lower gastrointestinal tract (Oakland et al. (2017)) and 45% of patients with presumed bleeding from the upper gastrointestinal tract (Stanley et al. (2017)) require urgent medical intervention; overall mortality rates for AGIB in the U.S. are estimated at around 3 per 100, 000 (Peery et al. (2022)). For patients who present with AGIB in the emergency room, the attending physician is tasked with deciding whether the bleeding is severe enough to warrant admission to the hospital. However, the specific etiology of AGIB is often difficult to determine from patient presentation alone, and gold standard diagnostic techniques an endoscopy for upper GI bleeding or a colonoscopy for lower GI bleeding - are both invasive and costly, particularly when performed urgently in a hospital setting. To aid emergency room physicians in making this determination more efficiently, the Gaaasgo--Blatchffe
Bleeding Score or GBS (Blatchford et al. (2000)) is a standard screening metric used to assess the risk that patient with acute upper GI bleeding will require red blood cell transfusion, intervention to stop bleeding, or die within 30 days. It has been also validated in patients with acute lower gastrointestinal bleeding' assess need for intervention stop bleeding or risk of death (A (Asad Ur-Rahman and Abusaada (2018)); accordingly, we interpret the GBS as measure of risk for patients who present with either upper or lower GI bleeding in the emergency department.
Construction of the Blasgow-Blatchhfe Score. The Glasgow- Blatchofod Score is function of the following nine patient characteristics: blood urea nitrogen (BUN), hemoglobin (HGB), systolic blood pressure (SBP), pulse, cardiac failure, hepatic disease, melena, syncope and biological sex. The first four are real-valued and the latter five are binary. The GBS is calculated by first converting the continuous features to ordinal values (BUN and HGB to 6 point scales, SBP to point scale and pulse to a binary value) and then summing the values of the first 8 features. Biological sex is used inform the conversion of HGB to an ordinal value. Scores are integers ranging from 0 to 23,
US and international guidelines use the Glaggow-Blatchford score as the preferred risk score for assessing patients with upper gastrointestinal bleeding (Laine et al. (2021); Barkun et al. (2019)). Other risk scores tailored to bleeding in the lower gastrointestinal tract have been proposed in the literature, but these are less commonly used in practice. We refer interested readers to Almaghrabi et al. (2022) for additional details.
7

---

with higher scores indicating higher risk that patient will require subsequent intervention. US and international guidelines suggest that patients with a score of 0 or 1 can be safely discharged from the emergency department (Laine et al. (2021); Barkun et al. (2019)), with further investigation to be performed outside the hospital. For additional details on the construction of the GBS, we refer to Blatchford et al. (2000).
Defining features, predictions and outcomes. We consider a sample of 3617 patients who presented with AGIB at one of three hospitals in a large academic health system between 2014 and 2018. Consistent with the goals of triage for patients with AGIB, we record an 'adverse outcome' if a patient (1) requires some form of urgent intervention to stop bleeding (endoscopic, interventional radiologic, or surgical; excluding patients who only undergo diagnostic endoscopy or colonoscopy) while in the hospital (2) dies within 30 days of their emergency room visit or (3) is initially discharged but later readmitted within 30 days.* As is typical of large urban hospitals in the United States, staffing protocols at this health system dictate a separation of responsibilities between emergency room physicians and other specialists. partieular, while emergency room physicians make an initial decision whether to hospitalize patient, it is typically gastrointestinal specialist who subsequently decides whether a patient admitted with AGIB requires some form of urgent hemostatic intervention. This feature, along with our ability to observe whether discharged patients are subsequently readmitted, substantially mitigates the selective labels issue that might otherwise occur in this setting. Thus, consistent with clinical and regulatory guidelines to avoid hospitalizing patients who do not require urgent intervention (Stanley et al. (2009)), and to avoid discharging patients who are likely to be readmitted within 30 days (NEJM (2018)) to admit or discharge a patient as a prediction that one of these adverse outcomes will occur. We model by letting X; E [0, 1]8 be the nine discrete patient characteristics from thus instantiate Bllacchforr Score is computed"; the only transformation we apply is to normalize Glasgow-Blatchforrd
interpret the emergency room physician's 's decision
which the Glasgow-BB
each feature to lie in [0, 1]. We further let & E {0, 1} indicate whether that patient was initially hospitalized, and Y; E {0, 13 indicate whether that patient suffered one of the adverse outcomes defined above.
Assessing the accuracy of physician decisions. We first summarize the performance of the emer- gency room physicians' hospitalization decisions, and compare them with the performance of a simple rule which would instead admit every patient with GBS above certain threshold and discharge the remainder (Table 1). We consider thresholds of and - the generally accepted range for low risk patients (Laine et al. (2021); Barkun al. (2019)) as well as less conservative thresholds of 2 and 7 (the latter of which we find maximizes overall accuracy). For additional context, we also provide the total fraction of patients admitted under each decision rule.
Decision Rule
Fraction Hospitalized
Accuracy
Sensitivity
Specificity
Physician Discretion
0,86 = 0.02
0.55 # 0.02
0.99 0.00
0.24 + 0.02
Admit GBS > 0
0.88 0.02
0.53 = 0.02
0.99
0.19 + 0.02
Admit GBS > 1
0.80 + 0.02
0.60 = 0.02
0.98 # 0.00
0.33 + 0.02
Admit GBS > 2
0.73 0.02
0.66 + 0.02
0.97 0.00
0.43 # 0.02
Admit GBS > 7
0.40 + 0.02
0.79 = 0.02
0.73 + 0.02
0.84 + 0.02
Table 1: Comparing the accuracy of physician hospitalization decisions ("Physician Discretion") ') to those made by thresholding the GBS. For example, 'Admit GBS > 1' hospitalizes patients with GBS strictly larger than 1. 'Fraction Hospitalized" indicates the fraction of patients hospitalized by each rule. 'Accuracy' indicates the 0/1 accuracy of each rule, where decision is correct if it hospitalizes a patient who suffers an adverse outcome (as defined above) or discharges a patient who does not. 'Sensitivity" indicates the fraction of patients who suffer an adverse outcome that are correctly hospitalized, and 'Specifiiityy' indicates the fraction of patients who do not suffer an adverse outcome that are correctly discharged. Results are reported 22 standard errors.
*This threshold is consistent with the definition used in the Centers for Medicare and Medicaid Services Hospital Readmission Reduction Program, which seeks to incentivize healthcare providers avoid discharging patients who will readmitted within 30 days (NEJM (2018)) 'ee consider alternative definitions of the feature space in Appendix E, where we find substantively similar results.
8

---

Unsurprisingly, we find that the physicians calibrate their decisions to maximize sensitivity (minimize false negatives) at the expense of admitting a significant fraction of patients who, in retrospect, could have been discharged immediately. Indeed we find that although 86% of patients are initially hospitalized, only z 42% actually require a specific hospital-based intervention or otherwise suffer an adverse outcome which would justify hospitalization (i.e., y4 = 1). Consistent with Blatchford et al. (2000) and Chatten et al. (2018), we also find that thresholding the GBS in the range of [o, 2] achieves sensitivity of close to 100%. We further can see that using one of these thresholds may achieve overall accuracy (driven by improved specificity) which is substantially better than physician discretion. Nonetheless, seek to test whether physicians demonstrate evidence of expertise in distinguishing patients with identical (or nearly identical) scores.
Testing for physician expertise. We now present the results of running ExpertTest for K = 1000 resampled datasets and various values of (where 1808 the largest possible choice given 3617) 7) in Table 2. We define the distance metric m((21222) = || 22|12, though this choice is inconsequential when the number of "mismatched pairs' (those pairs x; x' where x o x') O.
We also observe that, in the special case of binary predictions and outcomes, it is possible to analytically determine the number of swaps which increase or decrease the value of nearly any natural loss function (-)). Thus, although we let F(D)== == E [w; + ū.] for concreteness, our results are largely insensitive to this choice; in particular, they remain the same when false negatives and false positives might incur arbitrarily different costs. We elaborate on this phenomenon in Appendix E.
L mismatched pairs
swaps that increase loss
swaps that decrease loss
100
0
5
1
0.061
250
12
0.003
500
0
21
2 <.001
1000
42
2 <.001
1808
265
66
4 <.001
Table 2: The results of running ExpertTest, where each pair of patients is chosen to be as similar as possible with respect to the nine (discrete) inputs to the Glasgow- Blatchford score. L indicates the number of pairs selected for the test, of which 'mismatched pairs' are not identical to each other. Swaps that decrease (respectively, increase) loss indicates how many of the L pairs result in a decrease (respectively, increase) in the 0/1 loss when their corresponding hospitalization decisions are exchanged with each other. T is the p-value obtained from running ExpertTest.
As the results demonstrate, there is very strong evidence that emergency room physicians incorporate information which is not available to the that physicians can reliably distinguish patients who appear identical with respect the nine features considered by the GBS- and can make hospitalization decisions accordingly even though simple GBS thresholding is highly competitive with physician performance. This implies that no predictive algorithm trained on these nine features, even one which is substantially more complicated than the GBS, can fully capture the information that physicians use to make hospitalization decisions.
score. In particular, our test indicates
To interpret the value of T. observe that for L N 500 we recover the smallest possible value 1/(K + 1) = 1/1001. Furthermore, for all but the final experiment, the number of mismatched pairs is O, which means there is no additional type one error incurred (i.e., e (11) is guaranteed to be 0). For additional intuition on the behavior of ExpertTest, we refer the reader to the synthetic experiments in Appendix F.
6 Discussion and limitations
In this work we provide simple test to detect whether a human forecaster is incorporating unobserved information into their predictions, and illustrate its utility in a case study of hospitalization decisions made by emergency room physicians. A key insight is to recognize that this requires more care than simply testing whether the forecaster outperforms an algorithm trained on observable data; indeed, a large body of prior work suggests that this is rarely the case. Nonetheless, there are many settings in which we might expect that an expert using information or intuition which is difficult to replicate with predictive model.
9

---

An important limitation of our approach is that we do not consider the possibility that expert forecasts might inform decisions which causally effect the outcome of interest, as often the case in practice. We also do not address the possibility that the objective of interest is not merely accuracy, but perhaps some more sophisticated measure of utility (e.g., one which also values fairness or simplicity); this explored in Rambachan (2022). We caution more generally that there are often normative reasons to prefer human decision makers, and our test captures merely one possible notion of expertise. The results of our test should thus not be taken recommending the automation of given forecasting task.
Furthermore, while our framework is quite general, it is worth emphasizing that the specific algorithm we propose is only one possible test of Ho- Our algorithm does not scale naturally to settings where X' is high-dimensional, and in such cases it is likely that a more sophisticated test of conditional independence (e.g. kernel-based method; see Fukumizu et al. (2004), Gretton et al. (2007) and Zhang et al. (2011), among others) would have more power to reject Ho. Another possible heuristic is to simply choose some learning algorithm to estimate (e.g.) B[y X] and E[Y X, YJ. and examine which of the two provides better out of sample performance. This can be viewed as a form of feature selection; indeed the 'knockoffs' approach of Candès et al. (2016) which inspires our work is often used as a feature selection procedure in machine learning pipelines. However, most learning algorithms do not provide p-values with the same natural interpretation describe in section 3, and we thus view these approaches as complementary to our own.
Finally, our work draws clean separation between the 'upstream' inferential goal of detecting whether a forecaster is incorporating unobserved information and the 'downstream' algorithmic task of designing tools which complement otherwise incorporate human expertise. These problems share a very similar underlying structure however, and we conjecture that - as has been observed in other supervised learning settings, e.g. Kearns et al. (2018) 8) there is tight connection between these auditing and learning problems. We leave an exploration of these questions for future work.
References
Agrawal (2019). The economics of artificial intelligence. (NBER) National Bureau of Economic Research Conference Reports. University of Chicago Press, Chicago, IL.
Agrawal, A., Gans, J., and Goldfarb, A. (2018). Exploring the impact of artificial intelligence: Prediction versus judgment. Technical report, National Bureau of Economic Research, Cambridge, MA.
Almaghrabi, M., Gandhi, M., Guizzetti, L., lansavichene, A., Yan, B., Wilson, A., Oakland, K., Jairath, V., and Sey, M. (2022). Comparison of risk scores for lower gastrointestinal bleeding: A systematie review and meta-analysis. JAMA Netw: Open, 555)e2224
Amold, D., Dobbie, W., and Hull, P. (2020). Measuring racial discrimination in bail decisions. Technical report, National Bureau of Economic Research, Cambridge, MA.
Asad Ur-Rahman, Jian Guan, S. K. A. M. M. S. F. X. H. A. M. and Abusaada, (2018). Both full glasgow-blas blatch-d score and modified glasgow- score predict the need for intervention and mortality in patients with acute lower gastrointestinal bleeding. Digestive Diseases and Sciences, 63:3020-3025.
Bansal, G., Wu, T. S., Zhou, J., Fok, R., Nushi, B., Kamar, E.,, Ribeiro, M. T., and Weld, D. S. (2020). Does the whole exceed its parts? the effect of ai explanations on complementary team performance. Proceedings of the 2021 CIll Conference on Human Factors in Computing Systems.
Barber, R. F. and Candès, E. (2019). On the construction of knockoff's in case-control studies. Stat, 8(1): c225 sta4.225.
Barber, R. F,, Candès, F. J., and Samworth, J. (2018). Robust inference with knockoffs.
Barkun, A. N., Almadi, M., Kuipers, E. J., Laine, L.,, Sung, J., Tse, F., Leontiadis, G. I., Abraham, N. S., Calvet, X., Francis K.L. Chan, J. D., Enns, R., Gralnek, I. M., Jairath, V., Jensen, D., Lau, J., Lip, G. Y., Loffroy, R., Maluf-Filho, F., Meltzer, A. C., Reddy, N., Saltzman, R., Marshall, J. K., and Bardou, M. (2019). Management of nonvariceal upper gastrointestinal
10

---

bleeding; Guideline recommendations from the international consensus group. Annals of Internal Medicine, 171(11):805-822 822. PMID: 31634917.
Bastani, H., Bastani, O., and Sinchaisri, W. P. (2021). Improving human decision makiing with machine learning.
Berrett, T. B., Wang, Barber, F., and Samworth, (2018). The conditional permutation test for independence while controlling for confounders.
Blatchford, O., Murray, W. R., and Blatchford, M. (2000). A risk score predict need for treatment for upper-gastroointssinna haemorhage. Lancet, 356(9238):311813211
Camerer, C. and Johnson, E. (1991). The process-performance paradox in expert judgment: How can experts know so much and predict so badly? Ericsson, A. and Smith, J., editors, Toward General Theory of Expertise: Prospects and Limits. Cambridge University Press.
Candès, E., Fan, Y., Janson, L., and Lv, J. (2016). Panning for gold: Model-x knockoffs for high-dimensional controlled variable selection.
Cen, S. H. and Shah, D. (2021). Regret, stability fairness in matching markets with bandit learners.
Chatten, K., Purssell, H., Banerjee, K., Soteriadou, S., and Ang, Y. (2018). Glasgow blatchford score and risk stratifications in acute upper gastrointestinal bleed: can we extend this to 2 for urgent outpatient management? Clin. Med., 18-2):11--22..
Cowgill, B. (2018). Bias and productivity in humans and algorithms: Theory and evidence from resume screening.
Currie, J. and MacLeod, W. (2017). Diagnosing expertise: Human capital, decision making, and performance among physicians. J. Labor Econ., 35(1):1-45
Dawes, R. M. (1971). A case study of graduate admissions: Application of three principles of human decision making. Am. Psychol., 26(2):180-188)
Dawes, R. M., Faust, D., and Meehl, P. E. (1989). Clinical versus actuarial judgment. Science, 243(4899):1668-1677,
Dawid, A. P. (1979). Conditional independence statistical theory. Journal of the Royal Statistical Society, Series B (Methodological), 41(1):1-31.
Edmonds, J. (1965). Paths, trees, and flowers. Canadian Journal of mathematics, 17:449-667.
Eren, O. and Mocan, N. (2018). Emotional judges and unlucky juveniles. Am. Econ. J. Appl. Econ., 10(3):171-205.
Fukumizu, K., Bach, R., and Jordan, M. I. (2004). Dimensionality reduction for supervised learning with reproducing kernel hilbert spaces. J. Mach. Learn. Res., 5:73-99.
Gretton, A., Fukumizu, K., Teo, H., Song. L., Schölkopf, B., and Smola, A. (2007). A kernel
statistical test of independence. In Advances in Neural Information Processing Systems 20, pages 585-592 Curran Associates, Inc.
Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., and Nelson, C. (2000). Clinical versus mechanical prediction: meta-analysis. Psychol Assess, 12(1):19-30.
Hardt, M., Megiddo, N., Papadimitriou, C., and Wootters, M. (2015). Strategic classification.
Kearns, M., Neel, S., Roth, A., and Wu, Z. S. (2018). Preventing faimess gerrymandering: Auditing and learning for subgroup fairness. In Dy, J. and Krause, A., editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2564 2572. PMLR.
Keswani, V., Lease, M., and Kenthapadi, K. (2021). Towards unbiased and accurate deferral to multiple experts. Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society.

---

Kleinberg, J., Lakkaraju. H., Leskovec, J., Ludwig, J., and Mullainathan, S. (2017). Human decisions and machine predictions.
Kleinberg, J., Mullainathan, S., and Raghavan, M. (2022). The challenge of understanding what users want: Inconsistent preferences and engagement optimization.
Kleinberg, J. and Raghavan, M. (2018). How do classifiers induce agents to invest effort strategically?
Kuncel, N. R., Klieger, D. M., Connelly, B. S., and Ones, D. S. (2013). Mechanical versus clinical data combination in selection and admissions decisions: a mntaa-nnllysis. J Appl Psychol, 98(6):1060- 1072.
Laine, L., Barkun, A. N., Saltzman, J. R., Martel, M., and Leontiadis, G. L. (2021). ACG clinical guideline: Upper gastrointestinal and ulcer bleeding. Am. Gastroenterol., 116(5):899-99)
Lakkaraju, H., Kleinberg, J., Leskovec, J., Ludwig, J., and Mullainathan, S. (2017). The selective labels problem. In Proceedings of the 23rd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, New York, NY, USA. ACM.
Liu, L. T., Ruan, R. Mania, H., and Jordan, M. I. (2020). Bandit learning in decentralized matching markets.
Madras, D., Pitassi, T., and Zemel, R. (2018). Predict responsibly: Improving fairness and accuracy by learning to defer. In Advances in Neural Information Processing Systems 31, pages 6150-6160.
Mozannar, H. and Sontag, D. A. (2020). Consistent estimators for learning to defer to an expert. In International Conference on Machine Learning.
Mullainathan, and Obermeyer, Z. (2019). Diagnosing physician error: A machine learning approach to low-value health care. Technical report, National Bureau of Economic Research, Cambridge, MA.
NEJM (2018). Hospital readmissions reduction program (hrrp). Catalyst Carryover, 4(2).
Oakland, K., Jairath, V., Uberoi, R., Guy, R., Ayaru, L.,, Mortensen, N., Murphy, M. F., and Collins, G. S. (2017). Derivation and validation of novel risk score for safe discharge after acute lower gastrointestinal bleeding: a modelling study. The Lancel Gastroenterology Hepatology, 2(9):635-643.
Peery, A. F. Crockett, S. D., Murphy, C. C., Jensen, E. T., Kim, H. P., Egberg, M. D., Lund, J. L., Moon, A. M., Pate, V., Barnes, E. L., Schlusser, C. L., Baron, T. H., Shaheen, N. J., and Sandler, R. S. (2022). Burden and cost of gastrointestinal, liver, and pancreatic diseases in the united states: Update 2021. Gastroenterology, 162(2):621-644,
Perdomo, J., Zrnic, T., Mendler-Dünner, C., and Hardt, M. (2020). Performative prediction. In III, H. D. and Singh, A., editors, Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 7599-7609, PMLR.
Raghu, M., Blumer, K., Corrado, G., Kleinberg, J., Obermeyer, Z., and Mullainathan, S. (2019). The algorithmic automation problem: Prediction, triage, and human effort.
Rambachan, A. (2022). Identifying prediction mistakes in observational data.
Rastogi, C., Leqi, L., Holstein, K., and Heidari, H. (2022). unifying framework for combining complementary strengths of humans and ml toward better predictive bociisonnmaking.
Runge, J. (2017). Conditional independence testing based on a nearest-neighhor estimator of conditional mutual information.
Sen, R., Suresh, A. T., Shanmugam, K., Dimakis, A. G., and Shakkottai, (2017). -owee-p conditional independence test. In Advances in Neural Information Processing Systems 30, pages 2951-2961
Shah, R. D. and Peters, J. (2018). The hardness of conditional independence testing and the generalised covariance measure.
12

---

Stanley, A. J., Ashley, D., Dalton, H. R., Mowat, C.. Gaya, D. R., Thompson, E., Warshow, U., Groome, M.. Cahill, A., Benson, G., Blatchford. O., and Murray, W. (2009). Outpatient man- agement of patients with low-risk upper-gasaeeee sapper-gattoontestinnll haemorrhage: multicentre validation and prospective evaluation. Lancet, 373(9657):42-47.
Stanley, A. J., Laine, L., Dalton, H. R., Ngu, J. H., Schultz, M., Abazi, R., Zakko, L., Thornton, S., Wilkinson, K., Khor, C. J. L., Murray, I. A., and Laursen, S. B. (2017). Comparison of risk scoring systems for patients presenting with upper gastrointestinal bleeding: international multicentre prospective study. BMJ, 356.
Tversky, A. and Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185(4157):1124-11
Zhang, K., Peters, J., Janzing, D., and Scholköpf, B. (2011). Kernel-based conditional independence test and application in causal discovery. Proceedings of the Sweety-Sevetth Conference on Uncertainty in Artificial Intelligence, UAI'11, page 804 813, Arlington, Virginia, USA. AUAL Press.
13

---

A Additional related work
Human decision making, interplay with algorithms. Our work contributes to a vast literature on understanding how humans, and particularly human experts, make decisions. We do not attempt to provide comprehensive summary of this work, but refer the reader to Tversky and Kahneman (1974) and Camerer and Johnson (1991) for general background. Of particular relevance for our setting is work which investigates whether humans make systematic mistakes in their decisions, which has been studied in the context of bail decisions (Kleinberg et al. (2017), Rambachan (2022). Lakkaraju et al. (2017) and Amold et al. (2020)), college admissions (Kuncel et al. (2013), Dawes (1971)) and patient triage and diagnosis (Currie and MacLeod (2017), Mullainathan and Obermeyer (2019)) among others. One common theme in these works is that the decision made by the human expert will often influence the outcome of interest; for example, an emergency room doctor's ' initial diagnosis will inform the treatment patient receives, which subsequently affects their health outcomes. Furthermore, it is often the case that even observing the outcome of interest is contingent on the human's 's decision: for example, in college admissions setting, we might only observe historical outcomes for admitted students, which makes it challenging to draw inferences about applicants. This one-sided labeling problem is form of endogeneity which has been well studied in the context of causal inference, and these works often adopt causal perspective to address these challenges.
As discussed in Section 6, our instead work assumes that all outcomes observable and, importantly, that they are not affected by the human predictions. We also do not explicitly grapple with whether the human expert has objective other than maximizing accuracy under known metric (e.g., squared error). Though this is often a primary concern in many high-stakes settings - for example, ensuring that bail decisions are not only accurate but also nondiseriminatory it is outside the scope of our work, and we refer the reader instead to Rambachan (2022) for further discussion.
As discussed in section 1, another closely related theme is directly comparing human performance to that of an algorithm (Cowgill (2018), Dawes et al. (1989), Grove et al. (2000),, and developing learning algorithms which are complementary to human expertise (Madras et al. (2018), Raghu et al. (2019), Mozannar and Sontag (2020), Keswani et al. (2021), Agrawal et al. (2018) and Bastani et al. (2021)). A key design consideration when designing algorithms to complement human expertise involves reasoning about the ways in which humans may respond to the introduction of an algorithm, which may be strategic (e.g. Kleinberg and Raghavan (2018), Perdomo et al. (2020), Cen and Shah (2021), Hardt et al. (2015), Liu et al. (2020)) or subject to behavioral biases (Kleinberg et al. (2022)). These behaviors can make it challenging design algorithms which work with humans to achieve the desired outcomes, as humans may respond to algorithmic recommendations or feedback in unpredictable ways.
Conditional independence testing. We cast our setting as a special case of conditional independence testing, which has been well studied in the statistics community. For background we refer the reader to Dawid (1979). It has long been known that testing conditional independence between three (possibly high-dimensional) random variables is a challenging problem, and the recent result of Shah and Peters (2018) demonstrates that this is in fact impossible in full generality. Nonetheless, there are many methods for testing conditional independence under natural assumptions; perhaps the most popular are the kernel-based methods introduced by Fukumizu et al. (2004) and subsequently developed in Gretton et al. (2007) and Zhang et al. (2011), I), among others.
Our work instead takes inspiration from the 'knockoffs' framework developed in Candès et al. (2016), Barber et al. (2018) and Barber and Candès (2019), as well as the closely related conditional permutation test of Berrett et al. (2018). These works leverage the elementary observation that, under the null hypothesis that (specialized to our notation) the outcome Y and prediction Ý are independent conditional on the observed data X, new samples from the distribution of Y XX should be exchangeable with Y. Thus, if we know or can accurately estimate the distribution of Ý X, is straightforward to generate fresh samples ("knockoffs') which are statistically indistinguishable from the original data under the null hypothesis Hg : Y 1L Y X. Thus, if the observed data appears anomalous with respect to these knockoffs, this may provide
basis on which to reject Ho-
Our work avoids takes inspiration from this framework, but avoids estimating the distribution of Ý | X by instead leveraging a simple neireot-neighhors style algorithm for generating knockoffs. In that sense, our technique builds upon the wearest-neighbors based estimator of Runge (2017), and is
14

---

nearly identical to the one mee-narest-neighbo procedure proposed in the independence test of Sen et al. (2017). This algorithm is subroutine in their more complicated end-to-end procedure, which involves training model to distinguish between the observed data and knockoff's generated via swapping the "predictions' (again specializing their general test to our setting) associated with instances which are as close as possible under the l2 norm. By contrast, we analyze a similar procedure under different smoothness assumptions which allow us to recover p-values that are entirely model free.
conditional
B Proof of Theorem 1
We establish the proof of Theorem following the intuition presented in Section 3. Specifically, we first bound the type I error of Expert'est in the idealized case where the data set contains L identical pairs of observations x x'. We then refine this bound handle the case, which is more likely in practice, that the pairs chosen are merely close together. Our final bound thus includes additional approximation error to account for the 'similarity' of the pairs if we succeed in finding L pairs which are identical, we get nearly exact type I error control, whereas if we are forced to pair instances which are 'far apart', we incur additional approximation error. We formalize this intuition below.
An idealized bound. We first establish that IP(TK& S sa) + K+I for any a E [0,1] when x' for every (z, x") pair chosen by ExpertTest.
To that end, we observe n data points (z,, Vi. Ds). i E [n]. Let L = (bze-1 k2e : E [L]} denote the indices of the pairs chosen by ExpertTest, with 1i for l E [L] denoting the pairs themselves.
By assumption, ExpertTest succeeds in finding identical pairs:
Tiz--1 = Zize: V E [L].
(14)
Therefore, from the definition (9) it follows that r((z Di (Eiar )i.,)) = 1 for all & [L].
As discussed in Section 3, ExpertTest will repeatedly generate n fresh data points, denoted by D, as follows. For each index i E [m]] i.e. those not corresponding those selected in L pairs, we select exactly the observed data (I,, V/. D:).
For i E C. we sample (1 112e-1), (z.26, Yoze) be the observed values but sample the corresponding ÿ values from {122611, 02e), (fze, F26-1)} with equal probability. That is, we swap the ÿ values associated with (1i22-11, 3/2e--1) 1), (#rg2, V/22e) with probability 1. We argue that this resampling process is implicitly generating fresh, identically distributed dataset from the underlying distribution 'D conditioned on the following event F:
data triplet as follows: for i E (128-1, i2t), we let
F ((x,, Vis 01) i [m]]C]] U (1 : E L3 U ((0ge- Dize) V (WizerDize-)) 1): E [Z]]--
(15)
Why condition on F? As discussed in section 3, a straightforward test would involve simply resampling K fresh datasets from the underlying distribution Dx x Drjx Dyjx and observing that, by definition, these datasets are distributed identically to the observed data Do under Ho : Y IL Y X. While this would form the basis for a valid test along the lines of the one described in Section 3, it requires knowledge of the underlying distribution which we are unlikely to have in practice. Thus, we instead condition on nearly everything in the observed data - the values and exact ordering of X and the values and exact ordering of Y, and the values of ÿ up to a specific set of allowed permutations (those induced by swapping 0 or more paired Dize-1 9ige values). This substantially simplifies the resampling problem, as we only need to reason about the correct 'swap" probability for each such pair. This can be viewed an alternative factorization of the underlying distribution D under Ho- rather than sampling X Dx.Y Dyjx, Y Dyix- instead sample an event F Dy from the induced distribution over events of the form (15), and then sample Ý Dyy
First, we show that conditional on F, the resampled dataset D and the observed dataset Do are indeed identically distributed under Ho Y 1L Y X (that they are also independent, conditional on F, is
15

---

clear by construction). To see this, observe that for each & [L]:
I(((Fize-1 Mizze=11 Mize-1) (Tizer Vize: Dize))
(16)
= P P(Uigr-1 Tige 1 (BP(Dz Fige-1 ) P(Uiee Tize PT
Tige)
(17)
P ) Tize- (ūi(e Tize-1 (Paze P Tiize )TP (U Tibze)
(18)
P(( Mi28-1 Bize): (Iizes Miues Dize-1))
(19)
In above, (17) follows from Hg and the assumption that the data drawn i.i.d., and (18) follows
from assumption (14) that Tize Vige By construction, the events in (16) and (19) are the only
two possible outcomes after conditioning on F, and this simple argument shows that in fact they are equally likely.
Thus, let Di...., Dr be K independent and identically distributed datasets generated by the above procedure. Let Du be one additional sample from this distribution, which we showed was distributed identically to Du under the idealized assumption (14).
As discussed in Section 3, for any real-valued function F that maps each dataset to R, we have
TK E ( s F(Da)
(20)
where we use definition of 1[. S as in (7).
Because Du...., are ii.dd, and thus exchangeable, it follows that K Σ 1(F(D,) S F(D.)
is uniformly distributed {0, K 1}. Therefore, with a little algebra it can be verified that for any [0, 1]. TK satisfies
S a) a K+1
(21)
Because Do and Do are independent and identically distributed under (14), the same holds if we replace Do with Do- Thus, ExpertTest provides nearly exact type error control in the case that the idealized assumption (14) holds. This result will serve as a useful building block, as we'll now proceed to relax this assumption and bound the type error of ExpertTest in terms of the total variation distance between Do and Do-
Fixing the approximation. Di..... Dr are synthetically generated datasets that are independent and identically distributed. The argument above replaced the observed dataset Do with resam- pled 'idealized' dataset Do. which is also independent and identically distributed with respect to D....., Dx, and then used this fact to demonstrate that IP Do....Dx!> S a) S a K+I. If the idealized assumption (14) holds, replacing Do with Do is immaterial as we showed the two are identically distributed conditional on F. Of course, this assumption will not hold in general, and this is what we seek to correct next.
Let Do D.1r" be random variable distributed according to the true underlying distribution D, conditional on the event F. The observed data Do can be interpreted as one realization of this random variable. One way to quantify the excess type error incurred by using Do in place of Do to bound the total variation distance between the joint distributions of (Do.... Dx) and that of (Do, Di.... Dx). Specifically, follows from the definition of total variation distance that:
Dx P Dx (TK S a) < P D (= S a) + TV(IP TV(PP Dx Dx Do....DD
(22)
where TV(PP D TP Do.....DxIII 1 denotes the total variation distance between its arguments. Due to the independence of the resampled datasets, this simplifies
TV(P Dx.....D.F... Po....
Dr. DR( = TV(P DoIF; PAL,, (.)1F)
(23)
16

---

Therefore, we need only bound the total variation distance between P Doj and IP "ol. to conclude the proof.
As defined in (11), the e. provides us with a way of bounding the total variation distance between the distribution of Du and Do. To see this, observe that the distributions of Do and Da, conditioned on F, can be described as follows. To construct Du. we can imagine flipping L fair coins to decide the assignment Ys in each of the ( pairs; if it comes up heads, we swap the observed pair (iiz 2 2b22 and if it comes up tails we do not. The observed (Zz, M4) as well as yJ for i & L are set in Do they are observed in Do.
Du is constructed similarly, but we instead flip a coin with bias (1+r((Te Disy ), (Zize: Dize )))- to decide the assignment of (fiz) again, heads indicates that we swap the observed ordering, and tails indicates that we do not.
By construction, the distributions of Do and Do are identical conditioned on F, as r((Fize-1 (( 1)) (Zige )) denotes the true relative odds of observing each of the two possible (z, 1) pairings. In contrast, the distribution of Do is different, as it was sampled using the simplifying assumption (14) in particular, Do is generated assuming r((Fize-1. 1.00ee- 1i (Fizer Dize)) = 1!
The difference between the biases of these coins is bounded above by 5 We'll use this observation, along with the following lemma, to complete the proof.
Lemma 3 (Bounding the total variation distance between i.idd. coin flips) Let i E [L] index sequence of i.i.d. coin flips u1... Un each with bias Po, and U1 ... Ut be a sequence of f.idd. coin flips with bias Then we can show:
TV((u) ... uL), (Vl... UL)) S - (1 max lp: -;1)
(24)
We defer the proof of lemma 3 Appendix D. This implies that the total variation distance between Do and Do is bounded above by 1 - (1 )² This, along with (21), (22) and (23) concludes the proof of Theorem 1.
Corollary 3.1 (Weaker type I error bound)
P(r s a)
L+
(25)
K +
Corollary 3.1 is weaker bound than the one given in Theorem 1, but is easier to interpret and manipulate. We will make use of this fact the following section; the proof is an immediate consequence of theorem and provided in Appendix D for completeness.
C Proof of Theorem 2
To establish theorem 2, we will argue that +t. goes to 0 at a rate of O(n à). This implies that, provided L = o(nz), ), the excess type I error established in theorem 1 is o(1) as desired. To do this, we first show that each pair (Zize.1 Dize) chosen by ExpertTest will be close under the la norm (lemmas 4 and 5 below). We then leverage the smoothness assumption (12) to demonstrate that this further implies that e concentrates around O. For clarity we state auxiliary lemmas inline, and defer proofs to Appendix D.
Finding pairs which are close under the Ea norm.
Let Mr. to be the set of matchings of size L on &1...tn i.e. each element of M,, is a set of L disjoint (xr, (x)) x²) pairs. Let m² be the 'optimal' matching satisfying:
m E arg min max M (x,²")
||
x 2'112.
(26)
"This technique is inspired by the proof of type I error control given for the Conditional Permutation Test in Berrett et al. (2018); Appendix A.2 of their work for details
17

---

That is, m minimizes the maximum distance between any pair of observations in mutually disjoint pairing of 2L observations, Let
max (m.)")mm)
|| - x' 2'112.
(27)
That is, the smallest achievable maximum &g distance over all matchings of size L. We'll first show that:
Lemma 4 (Existence of optimal matching)
1f [0, 1) for some d 2
d' = =0 n1
(28)
with probability 1.
That is, there exists matching of size at least such the maximum pairwise distance in this matching scales like O(n= Lemma 4 demonstrates the existence of a sizable matching in which the maximum pairwise distance indeed tends to 0.? We next demonstrate that this approximates the optimal matching, at the cost of a factor of 2 on L.
Lemma (Greedy approximation to the optimal matching)
max [E
|| x2
d2g
(29)
That is, the maximum distance between any of the pairs of observations chosen by our algorithm will be no more than the maximum such distance in the optimal matching of size 2L.
Corollary 5.1 For L S B. we have:
22--1 - x21122 O (0 [E
max
(30)
This follows immediately by invoking lemma 4 to bound the right hand side of lemma 5, Corollary 5.1 demonstrates that grows large, the maximum pairwise Eg distance between L greedily chosen pairs will to zero at rate of O (n=2) provided L S z. We now show that the smoothness condition (12) further implies that, under these same conditions, we recover the asymptotic validity guarantee (13).
From approximately optimal pairings to asymptotic validity.
With the previous lemmas in place, the proof of theorem straightforward. Plugging the smoothness condition (12) into the definition of the odds ratio (9) yields the following:
For all (x2--1, 1, 12c-1), (222,, 322).
1
r (( 1, (124, U24))
(1 Cl 2222 (1 + Cl 22ell2)²
(31)
Where C' > 0 is the same constant in the definition of the smoothness condition (12). Corollary 5.1 shows that || 22el12 O n (n=1). :). (31) immediately implies that Ei defined in (11), also
goes to zero at rate of 0 (n- Thus, if we take to be a constant and K - oc, the type I error given (10) can rewritten as
1
P(TK S Sa+ -ar--aaaa(0011-e -
(32)
K+1
1
Sat Satem, b K+1
(33)
=a =a+0 (2
(34)
"In principle, we could find this optimal matching by binary searching for d' using the non- bipariiat maximal matching algorithm of Edmonds (1965); for simplicity. our implementation uses greedy matching strategy instead.
18

---

Where (33) follows from corollary 3.1. If we instead allow L to scale like o(nà) (still taking
K + 0o), (33) implies:
P(TK 5 S o(1)
(35)
which concludes the proof of theorem 2.
D Proofs of auxiliary lemmas
Proof of Lemma 3.
Recall that one definition of the total variation distance between two distributions P and Q is to consider the set of couplings these distributions. In particular, the total variation distance can be cquivalently defined as:
TV(P.Q) = inf P(X Y)
(36)
(x,Y (x()))((P.))
Where C(s--) is the couplings on its arguments. Consider then the following straightforward
coupling on X = (uI...uz)) and Y = (11 Uz):: draw L random numbers independently and uniformly from the interval [0, 1]. Denote these by 61 CL. Let u, 1[c; S pil. and
= 1[c I qil. It's clear that X and Y are marginally distributed according to ...... and ς1 ..... 9L. respectively. Furthermore, the probability that u, o Us is lp: qi by construction. Thus we have:
- =1 - (1 - lp: asl) S 1 (1 max Ipt xi))²
(37)
This concludes the proof.
Proof Corollary 3.1.
In the preceding proof of lemma 3, observe that we could have instead written:
PP +Y) = U
{m + u.)
Σ pps 2: S L max (p;
gil
(38)
EE(r]
xE[r.]
union bound KE(T]]
Specializing this result to the definitions Do and Do (and, in particular, the definition of ,,,)) completes the proof.
Proof of Lemma 4.
Our proof will proceed via a covering argument. In particular, we cover the feature space [0, 1] with a of non-overlapping d-dimensional hypercubes, each of which has edge length < 1, and show that sufficiently many pairs (Ir, x²) must lie in the same 'small' hypercube. To that end, let C' {c1-.. .k) be set of hypercubes of edge length with the following properties:
Ve E C. [-b, + bja
(39)
Vc, E C,ene' =0 Vx Du, 3c E
(40)
(41)
Where Do the observed data. It's clear that such covering C' must exist, for example by arranging
.r...Cr C2 in a regularly spaced grid which cover [0, 1]d (though note that per condition (39), some of these 'small' hypercubes may extend outside [0, 1] if does not evenly divide 1). Such a covering may be difficult to index care must be exercised around the boundaries of each small hypercube; however, as we only require the existence of such a covering, we ignore these details. We now state the following elementary facts:
(1 26)d
|0| <
(42)
Vc C, E Ec, - 2'112 s bvd
(43)
19

---

Where (42) follows because the volume of each C. is b", and the total yolume of all such hypercubes cannot exceed the volume of the containing hypercube [-b, 1+ bld, which gives us an upper bound on the size of the cover C. Furthermore, (43) tells us that for any (x, which lie in the same 'small' hypercube e, we have || x2'12 < bvd.
Let nee = Hx; Dt c} denote the number of observations contained in each small hypercube C C.
Corollary 5.2 For any E C, there exist at least [ disjoint pairs (x,, x²) E c such that || a'Il2 bvd.
With these preliminaries in place, we'll proceed to prove lemma 4. To do this, we'll first state one additional auxiliary lemma.
Let Na.6 ==
64 2 ga an upper bound on the number non-overlapping 'small' hypercubes with
a"
edge length b which can fit into [0, al". We'll show for any >> O, with b = va. a = + 26, we have:
Lemma 6 (Pairwise distance in terms of packing number)
n 2N = 4
pairs satisfying ||x - x' 1/2 S
(44)
That is, the pairwise distance between the closest of pairs (half the observed data in total) can be written in terms of the appropriately parameterized covering number. We defer the proof of this lemma the following section. For now, we simply plug in the definition of Na and rearrange to recover:
1 Va
2 23Vd
2N 0,6 2
(45)
n 21+
vd
Recall that is the maximum distance between any pairs (x. x') contained in the small hypercube with edge length particular, it holds for Va"
= The preceding argument holds for all > which satisfy (45), so in
2 Vd
(46)
n 21+3
2" is the maximum pairwise distance corresponding to one possible matching on (xx 2²) pairs, so this further implies that there exists matching M of size such that:
vd n 21 21- 2n)
|| x'112
(
With probability 1. Thus, follows that the maximum distance between any pair in the optimal matching d' also satisfies:
d: -0 ( 2 Vd
n 21+ = 000nn )0(=)))
With probability 1, as desired. This establishes the existence of matching of up to L disjoint pairs (x,x') E [0, 1]d such that the maximum distance between any such pair scales like O
We also consider the case where instead of X' [0, 1]d instead have P (X [0. 2 1 - 8 for some 8 E (0, For example, this will capture the case where X is (appropriately re-centered and re-scaled) multivariate Gaussian. In this case, we provide corresponding high probability version of lemma 4.
20

---

Corollary 6.1 Suppose instead of X = [0,1] 1]". we have for some E (0, 1):
P(X 11 1
(47)
Define m (1 6) We can then show:
2 Vd
P
d'
S
21-c
(48)
21+1
That is, we can still achieve constant factor approximation the optimal matching in Lemma with probability that exponentially approaches 1.
Proof of Corollary 6.1
Define the of points which falls in [0, 1] follows:
Su (X; X2 10.
(49)
and
Y4g |
(50)
is clear that in this setting, the proof of lemma holds if we simply replace n with nip the realized number of observations which fall in [0, 1].. However, no is now random quantity which follows a binomial distribution with mean (1 6)n (recall that we assume (Iz, Vss ūt) are drawn iidd. throughout). Thus, all that remains is bound no away from O, which we can via simple Chernoff bound:
8211-00
P(nn $ (1 - 5)2n) Se
(51)
Thus, it follows that
P(mo 2 (1 (1- 8)) 2 - 61mm
(52)
Thus, we have shown no 2 m with the desired probability. clear that we only require lower bound on no to recover the result of Theorem 4, as additional observations which fall in [0, 1J" can only improve the quality of the optimal matching d'...
Proof of Lemma
We will show that the procedure in ExpertTest which greedily pairs the closest remaining pair of points L times will always be able choose least one of the pairs an optimal matching of size 2L. Intuitively, this is because each pair (x, x²) chosen by ExpertTest can only 'rule out' at most two pairs (I, x"), (a', x²") in any optimal matching of size 2L. Thus, our greedy algorithm for choosing L pairs can perform no worse than an optimal matching of size 2L, the sense of minimizing the maximum pairwise distance.
Let mzz be an optimal matching of size 2L in the sense of (26). Then suppose towards contradiction that:
max |12-1 T2 dīr. [E[L]
(53)
Where dūr, is the smallest achievable maximum distance for any matching of size 2L as in (27).
Finally, let Im arg min || x2il22 džr; i.c. the first pair which chosen by ExpertTest that violates (53). Because pairs are chosen greedily to minimize la distance, and mēs is a matching of size 2L where all pairs are separated by at most dÿ,, under the l2 norm, it must be that none of the pairs which make up mł, were available ExpertTest at the 4n iteration. In particular, at least one element of every (z, x') pair in m must have been selected on previous iteration:
v(z,z') E m2g (Z1.... ......2 12, 2f V V {21 ... F21..22
(54)
21

---

As may contains 2L disjoint pairs 4L observations total - this implies that 2 - 2L 7 In
-1 -
= 4m L. This is contradiction, as ExpertTest only chooses pairs, so mm only
ranges in [1, I... This completes the proof.
Corollary 6.2 Validity in finite samples
Theorem 2 implies that we can achieve bound the excess type ervor in finite samples if we knew the constant in (12). In particular, let
m max ||22-11 - #22/|2 rE[L]
(55)
max
(56)
KE (LL4+Cm))22200 2 r
2
Then (10) implies that we can always construct valid (if underpoweree) test exactly the nominal size O by updating our REJECT threshold to
a - - (1- K+1
Proof of lemma 6
let C (er...x denote any set 'small' nonoverlapping hypercubes of edge length satisfying properties (39), (40) and (41). As discussed in the proof of lemma 4, each element of C is not
guaranteed to lie strictly each element of the cover instead contained the slightly larger hypercube [-b, + bjd As the
[0, 1]. Rather, each E must merely intersect [0, 1]". implying that
proof of lemma 4, we'll again let n. denote the number of observations which lie in some c E C.
By Corollary 5.2, we have that [ pairs in each c E C will satisfy || - x'|12 S bv 2. Thus
what's left to show that:
2 2N 2Na
Σ
Rej
n
2
4
jE[M]
We can see this via the following argument:
Σ
nej
N
Rej
1
(57)
2
2
JE[K]
JE(M)
k
(58)
2
Na.b
2
(59)
2
71
11
(60)
4
4
Where (59) follows from (42) and the definition of No.s, and (60) follows because n 2 2N..5 by assumption. This completes the proof.
E
Omitted Details from Section 5
E.1
Identifying relevant patient encounters and classifying outcomes
As described in Section we consider set of 3617 patients who presented with signs or symptoms of acute gastrointestinal bleeding at the emergency department at a large quaternary academic hospital system from January 2014 to December 2018. These patient encounters were identified using database mapping with a standardized ontology (SNOMED-CT) and verified by manual physician chart review. Criteria for inclusion were the following: any text that identifies acute gastrointestinal bleeding for hematemesis, melena, hematochezia from either patient report physical exam findings
22

---

(which were considered equally valid for the purposes of inclusion). Exclusion criteria were the following: patients with other reasons for overt bleeding symptoms (e.g. epistaxis) or missingness in input variables required to calculate the Glasgow- -aatch--f Score. As mentioned in Section 5, only upper gastrointestinal bleeding guidelines recommend the Glasgow-Blatchfon Score has also been validated for use in patients with
Score for routine
clinical use. However, the Glasgow-Blltchiee lower gastrointestinal bleeding, and in our setting the GBS was applied to patients who presented with signs of symptoms of either upper or lower gastrointestinal bleeding, We refer the reader to Section 5 for additional details and references.
This identified set of 3627 patients, of which further 10 were removed from consideration due to unclear emergency department disposition (neither Admit nor Discharge). As described in Section 5, we record an adverse outcome (Y = 1) for admitted patients who required some form of hemostatic intervention (excluding diagnostic endoscopy or colonoscopy), or patients who are readmitted or die within 30 days. We record an outcome of 0 for all other patients.
The use of readmission as part of the adverse event definition subject to two important caveats. First, we are only able to observe patients who are readmitted within the same hospital system. Thus, although the hospital system we consider is the dominant regional health care network, it is possible that some patients subseqeuently presented elsewhere with signs or symptoms of AGIB; such patients would be incorrectly classified as not having suffered an adverse outcome. Second, we only record an outcome of 1 for patients who are readmitted with signs or symptoms of AGIB, subject to the same inclusion criteria defined above. Patients who are readmitted for other reasons are not recorded as having suffered an adverse outcome.
E.2 The special case of binary outcomes and predictions
In our experiments we define the loss measure F(D)= 1 1/ + ūs). but it's worth remarking that this is merely one choice within a large class of natural loss functions for which ExpertTest pro- duces identical results when Y. Y are binary. In particular, observe that a swap of (v1, .n), (V2, 82) can only change the value of F(-) if y1 A 1/2 and ~ * $2 (we'll assume throughout that all observa- tions contribute equally to the loss; i.e. it is invariant to permutations of the indices i E [n)) This implies that there are only 2 out 2a possible configurations of (3/1, U1, 3/2 ÿ/2 where a swap can change the loss at all. Of these, two configurations create false negative and a false positive in the synthetic data which did not exist in the observed data:
(1/ = 1, 01 = 1, 9/2 = 0, 822 = =0 +
(111 = 1, 81 = 0, 1/2 = 0. 02 =
swap
original data
synthetic data
(91 0, أ1 0, 1/2 = 1, 0/2 1) swap
(y1 = 0, Φ1 1, 1/2 = 1, = 0)
The other two configurations which change the loss are symmetric, in that a swap removes both a false negative and false positive that exists in the observed data:
(3) 0, 01 1,
1, 1/102 0)
(y1 0, 91 0, 1/2 1, = 1)
swap
(y1 = 1, 0, = 0, 022 1)
(31 = 1, 91 = =1, 3/2 = 0, 02 0)
Thus, for any natural loss function which is strictly increasing in the number of mistakes E, 1 [M * Ø.]. the first two configurations of (g1, U1, V/, 92) will induce swaps which strictly increase the loss, while the latter two will induce swaps that strictly decrease the loss. This means that for given set of L pairs, we can compute the number of swaps which would increase (respectively, decrease) the loss for any function in this class of natural losses. In particular, this class includes loss functions which may assign arbitrarily different costs to false negatives and false positives. Thus, in the particular context of assessing physician triage decisions, our results are robust to variation in the way different physicians, patients or other stakeholders might weigh the relative cost of false negatives (failing to hospitalize patients who should have been admitted) and false positives (hospitalizing patients who could have been discharged to outpatient care).
23

---

E.3 Alternative feature spaces
In Section 5, we run ExpertTest by pairing patients who appear as close as possible (and. in most cases, identical) with respect to the nine features which are provided as input to the Glasgow- Blatchford score. While this test has perhaps the most natural interpretation in our setting, we could in general choose other ways of representing patients. In particular, a natural question to ask is whether physicians successfully distinguish patients who present with identical Glasgow-BAl Glasgow-Blatchford Blatchford scores (or, more generally, with identical predictions under some model of interest). That is, rather than letting X E [0, 1 be the nine patient characteristics which are provided as inpul to the GBS, we instead let X E (0. 1 23) be the GBS itself. While this is in some sense 'weaker' than the test we run in Section 5 the GBS is deterministic function of the richer feature space we chose there- recall that the validity of ExperfTest relies on finding pairs which are identical or nearly identical in the feature space. Thus, choosing to instead condition on a simple univariate prediction can in general yield useful insight when ExpertTest fails to offer conclusive results in the 'natural' input space. We present the results of this experiment in Table below.
L mismatched pairs
swaps that increase loss
swaps that decrease loss
100
4
0
0.043
250
0
11
2
0.004
500
0
21
3
<.001
1000
0
36
3 <,001
1808
71
5 <.001
Table 3: The results of running ExpertTest, where each pair of patients is chosen be as similar as possible with respect to their Glasgow-Blatchford scores. L indicates the number of pairs selected for the test, of which 'mismatched pairs' are not identical to each other. Swaps that decrease (respectively, increase) loss indicates how many of the L pairs result in decrease (respectively. increase) in the 0/1 loss when their corresponding hospitalization decisions are exchanged with each other. T the p-value obtained from running ExpertTest.
As we can see, we get very similar results to those presented in Section 5. Consistent with Table 2, we find no mismatched pairs for L < 1000, and we again obtain the smallest possible p- p-values of 1 for larger values of T.
K+I TooI
Another experiment we might run is to condition on a richer feature space than the one we chose in Section 5. In particular, recall that of the nine inputs to the GBS, four of them (blood urea nitrogen (BUN), hemoglobin (HGB), systolic blood pressure (SBP) and pulse) are real-valued test scores or vital signs. Part of the standard construction of the Glasgow Blatchhfrd Score entails converting these features to simple discrete scales, with larger values indicating that a patient is higher risk. A useful consequence of this convention is that it allows us find large numbers of patients who appear identical with respect to these discretized features. However, we can also run ExpertTest in the 'original' feature space, where each of these four characteristics take values in R and thus effectively guarantee that we fail to find pairs of patients who are exactly identical. We present the results of this test (where the only transformation we apply is to again rescale each feature lie in [o, 1) in Table below.
Here again strong evidence of expertise, with p-values equal K+1 1001 for larger values of L.. However, unlike in Table 2 and Table 3, we now fail to find identical pairs of patients for any value of L. This raises the possibility that ExpertTest is not valid in this setting, as type I eиror control now relies on the value of (11). While we cannot compute = without making assumptions about the underlying data distribution, we can provide a sense check by examining the Euclidian distance between each pair of patients. If it is very close to 0 for most or all pairs, we might reasonably expect that ExpertTest recovers approximate type I error control. We present these results in Figure 1 below.
As expected, we fail to find any pairs of patients who are identical in this partially real-valued feature space. Nonetheless, we see that for S 1000, every pair is chosen such that the Euclidian distance between them very close to O, which suggests that ExpertTest likely to be approximately valid for these experiments. We also see however that for L 1808, some pairs are chosen such that the
24

---

L
mismatched pairs
swaps that increase loss
swaps that decrease loss
100
100
5
0
0.015
250
250
13
2
0.005
500
500
23
3
<.001
1000
1000
36
6 <.001
1808
1808
66
6 <.001
Table 4: The results of running ExperTTest, where each pair of patients is chosen to be as similar possible with respect nine patient characteristics. The from these nine characteristics, after pre-processing step which discretizes each feature. indicates the number of pairs selected for the test, of which 'mismatched pairs' not identical to each other. Swaps that decrease (respectively, increase) loss indicates how many of the L pairs result in decrease (respectively, increase) in the 0/1 loss when their corresponding hospitalization decisions are exchanged with each other. T is the p-value obtained from running ExpertTest.
score is computed
Pairwise Distances in Original Feature Space
1808
1000
L
100
500
250
500
1000
1808
250
100
3
pairwise euclidian distance
Figure 1: Distribution of Euclidian distances between each pair of patients chosen by Ex- pertTest when patients represented as vector of nine patient characteristics, of which four blood urea nitrogen (BUN), hemoglobin (HGB), systolic blood pressure (SBP) and pulse are real-valued. L indicates the number of pairs of patients chosen for each experiment, with the boxplot indicating the distribution of pairwise Euclidian distances between them. The red line at v9 3 indicates the maximum possible Euclidian distance in this feature space.
pairwise distance is more than 30% of its maximum possible value, suggesting that ExpertTest does indeed incur substantial excess type I error when L 1808.
F Numerical Experiments
We first elaborate here on the example presented in the introduction. Consider the following stylized data generating process:
Example: experts can add value despite poor performance.
Let X, U, 61, Go be independent random variables distributed as follows:
X 2((-2,2), e ~ ~ ~N(10
25

---

Where u()) and (-: are the uniform and normal distribution, respectively. Suppose the true data generating process for the outcome of interest
Y = X+U+E
Suppose human expert constructs a prediction modeled as:
ÿ which is intended to forecast Y and can be
Y sign + sign(U) + E2
Where (X) = 1[X > 0] _[X < 0].
We compare this human prediction to that of an algorithm f(:) which can only observe X, and correctly estimates
f(x) F|Y X =
As described in the introduction, we use this example to demonstrate that ExpertTest can detect that the forecast y is incorporating the unobserved U even though the accuracy of Ý substantially worse than that of f(X). In particular, we consider the mean squared error (MSE) of each of these predictors:
Algorithm MSE =
f(x.))
71
Human MSE 1 n -0 81)2
We'll show below that the Algorithm MSE is substantially smaller than the Human MSE. However, we may also wonder whether the performance of the human forecast Y is somehow artificially constrained by the the relative scale of y and Y, as the sign(-) operation restricts the range of Y. For example, forecaster who always outputs Y 100 is perfectly correlated with the outcome but will incur very large squared error; this is special case of the more general setting where human forecasts directionally correct but poorly calibrated. To test this hypothesis, least squares regression (OLS) of Y on Y and compute the squared error of this rescaled prediction. is well known OLS estimates the optimal linear rescaling with respect to squared error, and we further use the in sample MSE of this rescaled prediction to provide lower bound the achievable loss. In particular, let:
ordinary
(8*,c")
= min Y J.CER
B8 - c
(61)
Rescaled Human MSE
(Y: c²)2
(62)
In Table 5 we report the mean squared error (plus/minus two standard deviations) over 100 draws of n 1000 samples from the data generating process described above. As we can see, both the original and rescaled human forecasts substantially underperform f(-).
Table 5: Expert vs Algorithm Performance
Algorithm MSE
Human MSE
Rescaled Human MSE
1.33 0.12
2.67 # 0.24
1.92 0.16
We now assess the power of ExpertTest in this setting by repeatedly simulating n = 1000 draws of (X, U, 62) along with the associated outcomes Y = X U é1 and expert predictions Ý = sign(X) (X) + (U) + c2- We sample 100 datasets this manner, and run ExpertTest on each one with L., = 100, and the distance metric m(x, x²) = / (x x')2 The distribution of p-values T1... is plotted in Figure 2.
26

---

1.0-
0.75
0.50
0.75
Figure distribution sharply nonuniform when the expert incorporates unobserved information U in the toy example. The vertical red line indicates critical threshold of a .05, and the dashed line traces a uniform distribution.
We that ExpertTest produces highly nonuniform distribution of the p-value and rejects the null hypothesis 94% of the time at critical value of a =005 To assess whether this power comes at the expense of an inflated type error, also run ExpertTest with both and U 'observed's in particular, suppose the distance measure was instead m((r, u). (a',u') v(x - x²)² + (u w/)² with everything else defined as above. The distribution of in this setting is again plotted in Figure 3.
0.00
0.55
0.50
075
4.00
Figure 3: distribution is approximately uniform when the expert does not incorporate unobserved information in the toy example. The vertical red line indicates a critical threshold of a .05, and the dashed line traces a uniform distribution.
When both X and V observed, and thus the null hypothesis should not be rejected, we instead that we instead get an approximately uniform distribution of r with false discovery rate of only .03 at critical value of .05. Thus, the power of ExpertTest to detect that the synthetic expert is incorporating some unobserved information U does not come at the expense of inflated type error, at least in this synthetic example.
Assessing the power of ExpertTest
We now present additional simulations to highlight how the power ExpertTest scales with the number of pairs L and the sample size in a more general setting, In particular, consider simple
27

---

synthetic dataset (2,, Ve. i E [n] f1.....1) where fi...n, = 2,,,, 21' and yi..../ the alternating binary string [0, 1, 0,. (we consider only even n for simplicity). This guarantees that each of the L pairs chosen are such that (z2e-1 z22) and y2c-1 + y2c. Importandy, it's also clear that x is uninformative about the true outcome - if the expert can perform better than random guessing, it must be by incorporating some unobserved signal U.
We model this unobserved signal by an 'expertise parameter' pair (124-1, U2e) for E [1... 21. we sample (U2e-1: f/ze) such that (f2e-1, 022) (U22-1: (22)) with and (y2e. 122-1) otherwise. Intuitively, 8 governs the degree which the expert predictions Y incorporate unobserved information - at = O, we model an expert who is randomly guessing, whereas at the expert predicts the outcome with perfect accuracy.
[0. In particular, for each
probability _
First, we consider the case of n E {200, 600, 1200} and fix L at as suggested by the proof of Theorem 2. For cach of these cases, examine how the discovery rate scales with the expertise In particular, we choose a critical threshold of a = .05 and
parameter E (5,.005...55.550))) compute how frequently ExpertTest rejects Hu over 100 independent draws of the data for each value 5. These results are plotted below in Figure 4.
1.00
Ledd
0.50
nHOCO, Le73
2e11000 L:150
8
Figure 4: The power of ExpertTest as a function of sample size n and expertise parameter 8. The horizontal dashed line corresponds to power of 80%
Unsurprisingly, the power of ExpertTest depends critically on the sample size at n 1200, ExpertTest achieves 80% power in rejecting Ho when the expert only performs modestly better than random guessing (6 As .1). In contrast, at n 200, ExpertTest fails achieve 80% power until 8 Rs .25 corresponding to an expert who provides the correct predictions over 75% of the time even when the observed x is completely uninformative about the true outcome.
Next we examine how the power of ExpertTest scales with L. We now fix = 600 and let 6 = .2 model an expert who performs substantially better than random guessing, but is still far from providing perfect accuracy. We then vary L {20, 40 200} and plot the discovery rate (again at critical value of a .05, over 500 independent draws of the data) for each choice of These results are presented below in Figure 5.
that power is monotonically increasing in L, and asymptotically approaching 1. With = 2, we see that ExpertTest achieves power in the neighborhood of only 50% with L 20 pairs, but sharply improves to approximately 80% power once L increases to 40. Beyond this threshold we see that there are quickly diminishing returns to increasing L.
As expected,
Excess type error of ExpertTest
Recall that, per Theorem 1, ExpertTest becomes more likely to incorrectly reject Ho L increases relative to n. In particular, larger values of will force ExpertTest to choose (ær, x²) pairs which are farther apart under any distance metric m(:.-), and thus induce larger values of = L. as defined in
28

---

1.04
0.75
0.50
100
Figure 5: The power of ExpertTest as a function of L, with n 600, 6 = .2. The horizontal dashed line corresponds to power of 80%
(1I). Furthermore, even for fixed r,, 0, the type one error bound given in Theorem I degrades with L. We empirically investigate this phenomenon via the following numerical simulation.
First, let X (X1, X2, R be uniformly distributed over [0, 10 Let Y X and Ý X1 + X2 + X3 + E2, where c1, E2 are independent standard normal random variables. In this setting, it's clear that Ho
Y X holds.
We repeatedly sample n 500 independent observations from this distribution over (X, and run ExpertTest for each L E {25, 50 250}. We let K = 50 and m(z, x') = || x'1 be the l distance. We let the loss function F-)) be the mean squared error of Y with respect to Y. For each scenario we again choose a critical threshold of = .05, and report how frequently ExpertTest incorrectly rejects the null hypothesis over 50 independent simulations in Figure 6.
Ialo diecavery
0.00
200
Figure 6: The type error rate of ExpertTest as function of L. with 500 and critical threshold of .5. The horizontal dashed line corresponds to the nominal false discovery rate of .05
As we can see, the type error increases sharply as function of L, and ExpertTest incurs false discovery rate of 100% at the largest possible value = 5! This suggests that significant care should be exercised when choosing the value of L, particularly in small samples, and responsible use of ExpertTest will involve leveraging domain expertise to assess whether the pairs chosen are indeed 'similar' enough to provide type I errr control.
29

---

G Pseudocode for ExpertTest
In this section we provide pseudocode for ExpertTest. Inputs Do. K, a, F()mm(;))) as defined in Section 3.
ExpertTest(Do: a, F().m,;;
Xo e {ar (x, E Do}
D initialize set of remaining observations initialize set of paired predictions
P P 0
for 1===111 do
(12)-1; - argmin (1 x')
>> find closest remaining pair, breaking ties arbitrarily
(x,z²)
\ 1220-1.320)
P-PUULE (1926-11,22))
D save predictions associated with closest remaining pair
end for
fo F(Do)
> calculate observed loss
for
1: do
D& < swap Do. P, 2) ft F(Dx)
independently swap each (f2r-1, E P with equal probability D calculate synthetic loss
end for
R Ex S fol
calculate quantile of observed loss, breaking ties random b if a, Ho is rejected with p-value + K+1
if T then REJECT end if
30


Reviews:
[{"review_id": "Wgzk24yHDn", "reviewer": "Reviewer_DMEm", "summary": "This paper proposes a hypothesis-testing approach to identify whether a set of predictions made by a human expert uses additional information that is conditionally independent from the input covariates. The paper provides theoretical guarantees for the test in a general setting and asymptotically. The proposed test is then applied to real-world data with emergency room physicians.", "questions": "Major:\n- Could you address the questions raised in the weaknesses mentioned above?\n\nMinor: \n- Notation confusion: U is used to both denote private information in Section 2 and random variables and observations in Section 3\n- What would the test return if the human is using information U that is *correlated* with X? Here, this may be a case where U is easier for a human to notice compared to X.\n- The table captions are not descriptive, which hinders readability.\n- Tone of contribution: the authors state that this test is a “necessary condition for achieving human-AI complementarity” without demonstrating results in terms of improving decision outcomes.\n", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper itself is written in a way that is easy to follow along, building the motivation and proposed work in a step-by-step manner.\n- The paper proposes a test that is a novel application of conditional independence work in the statistics literature for a novel use case. \n- The paper evaluates the theoretical test on a real-world use case.\n", "weaknesses": "- Can the test help improve decision outcomes? Typically, the primary goal in human-AI decision-making is to achieve complementarity (e.g., as discussed in [1]), particularly by leveraging the complementary skills of human and AI. Because the test does not account for human and AI prediction accuracies, it is difficult to say whether performing such a test has any implications on complementarity. Another relevant cite is [2].\n\n[1] Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. Bansal et al. CHI 2021.\n\n[2] A Unifying Framework for Combining Complementary Strengths of Humans and ML toward Better Predictive Decision-Making. Rastogi et al. EAAMO 2022.\n\n- Experimental validation: While it is great that the authors perform experiments on real-world data, it would be ideal to also verify the behavior of the test using synthetic where the differences between human and AI can be more carefully controlled to establish how sensitive the test is to these similarities and potentially the effect on the choice of L.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback! Our responses are provided below.\n\n**In response to**\n> \"Can the test help improve decision outcomes? Typically, the primary goal in human-AI decision-making is to achieve complementarity (e.g., as discussed in [1]), particularly by leveraging the complementary skills of human and AI. Because the test does not account for human and AI prediction accuracies, it is difficult to say whether performing such a test has any implications on complementarity. Another relevant cite is [2].\".\n\n1. This is an excellent question, and is addressed in Section 6 -- while the algorithm we give here is focused on the auditing problem, we hope that our framework yields insight into the structure of algorithms for Human/AI complementarity.\n2. While providing such algorithms is beyond the scope of our work, we respectfully disagree that \"it is difficult to say whether performing such a test has any implications on complementarity\". A rejection of our test indicates that the human forecaster *is* usefully incorporating information which is unavailable to any predictive algorithm, even if such information is incorporated in a suboptimal way. \n3. This suggests that, for example, *some* meta-algorithm which incorporates both human and baseline algorithmic forecasts should be able to achieve complementarity (given sufficient data, expressive power and other considerations which are standard in supervised learning tasks). On the other hand, a failure to reject our test suggests that *no* algorithm could hope to achieve complementarity, as the human is not incorporating useful information beyond that which is contained in the available features.\n4. Thus, we view our test as a precursor to assessing whether complementarity is likely to be achievable in a given forecasting task.\n\n**In response to**\n> \"Experimental validation: While it is great that the authors perform experiments on real-world data, it would be ideal to also verify the behavior of the test using synthetic where the differences between human and AI can be more carefully controlled to establish how sensitive the test is to these similarities and potentially the effect on the choice of L.\"\n\nThis concern is addressed in our response to all reviewers (see above).\n\n**In response to**\n> \"What would the test return if the human is using information U that is correlated with X? Here, this may be a case where U is easier for a human to notice compared to X.\"\n\nThis concern is addressed in our response to all reviewers (see above).\n\n**In response to other feedback**\n\nWe agree with the remaining suggestions, and thank you for your feedback. We will incorporate the two suggested references and change the notation in Section 3 to avoid clashing with Section 2. \n\n\n"}, {"review_id": "4XEH1fYtZM", "reviewer": "Reviewer_7cTX", "summary": "This paper proposes a method for determining whether a human expert is usefully using outside information that a model does not incorporate in order to make decisions. The goal is to test whether complementarity, humans working with models, is possible for a given task. The paper sets forth an algorithm, ExpertTest, provides some theoretical guarantees, and uses emergency room admissions as a case study for the technique. In the case study, the method indicates that doctors are making use of external information that is not captured in a commonly used risk score. \n", "questions": "1) Early in the paper, the authors equate $X$ with features, such as input features for a model. However, in the case study, the authors define $X$ as a GBS score, which is the output of a risk model. I think the notation needs to more carefully distinguish between what is an input to a model and what is an output of a model, and this may be part of the confusion that I express above in the weaknesses section. Could you clarify this point?\n\n2) In the case study, $\\hat{Y}$ is defined as whether the patient is admitted, and the ground truth $Y$ is defined as whether they suffered one of three adverse outcomes. Are these variables correlated? If a patient is admitted, are they less likely to suffer an adverse outcome? It seemed like this might actually be a setup where the “prediction” and “ground truth” are not independent of each other.\n\n3) The third adverse outcome mentioned is initial discharge and readmission within 30 days. Does this mean that anyone who suffered that outcome had a $\\hat{Y}$ of 0? I am just wondering if “initial discharge” means that the ER physician made the choice to not admit them. \n\n4) How sensitive is ExpertTest to the choice of loss function?\n", "limitations": "I think the paper should more carefully address the assumptions in the initial problem setup, specifically what I lay out in the weaknesses section. What model of human expertise is being considered when the assumption is that any additional information is encapsulated in the $U$ variables?\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main strength of this paper is the novel problem that it seeks to solve. Understanding whether and how humans can add their expertise on top of automated decision systems is an important goal, and it is often understudied. It is a creative approach to a crucial problem. \n", "weaknesses": "The primary weakness of the paper, in my opinion, is that the problem setup focuses on variables, or features, rather than also considering the functional form for the prediction itself. Fundamentally, if we are comparing the performance of something like the Glasgow-Blandford score (GBS) to humans, how do we know that the human is not using the same exact input features $X$ as the GBS, but just has a better way of mapping those $X$ to the prediction we are interested in? Why does $U$ have to exist at all for the human to be lending their “expertise” to the problem? That is, let’s say the GBS output is $\\tilde{Y} = \\tilde{f}(X)$, and the human output is $\\hat{Y} = \\hat{f}(X)$. If $\\hat{f}$ is closer to the actual generating function in the ground truth than $\\tilde{f}$, then the human could be “adding expertise” without actually using additional information as defined in this paper. Perhaps the human’s training would help them map these variables better than the GBS algorithm does. If I am missing something here, please correct me, but it’s very unclear to me why the method proposed leads to the conclusion that the expert is using additional features to make a decision. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback! Our responses are provided below.\n\n**In response to**\n> \"The primary weakness of the paper…is that the problem setup focuses on variables, or features, rather than also considering the functional form for the prediction itself.\"\n\n**We reproduced our original case study using the clinical features which make up the Glasgow-Blatchford Score. The results are attached with our response and discussed below.** We will include these results alongside our old ones in the final draft of Section 5. We respectfully disagree that the case study in our original draft is invalid however, and first provide a clarification:\n\n1. Recall that we seek to test whether the predictions $\\hat{Y}$ incorporate information other than the observable features $X$ to forecast an outcome $Y$. \n2. In Section 5, we instantiate this framework where $X$ is *just* the Glasgow-Blatchford score (GBS). That the GBS is itself a summary of other clinical markers **is merely provided as context. Our goal is to test whether physicians incorporate information other than the GBS itself**. \n3. You are absolutely correct that physicians may be relying on the *same* clinical markers but perhaps in a different way (i.e. by implicitly employing a different functional form). This is not a contradiction however -- the interpretation of ExpertTest simply depends on the set of features we choose to condition on. \n4. We agree however that it is helpful to see the results both conditional on the GBS itself and conditional on its constituent parts. Indeed, these results strengthen our work significantly. \n\n**Description of ‘input’ features**\n\nWe briefly contextualize the attached figures here, focusing on technical validity due to space constraints. Our final draft will include additional context regarding the clinical relevance of these features. The Glasgow-Blatchford Score is composed of the following nine features: blood urea nitrogen (BUN), hemoglobin (HGB), systolic blood pressure (SBP), pulse, cardiac failure, hepatic disease, melena, sycope and biological sex. The first four are continuous and the latter five are binary. The GBS is calculated by first converting the continuous features to ordinal values (BUN and HGB to 6 point scales, SBP to a 3 point scale and pulse to a binary value) and then summing the values of the first 8 features. Biological sex is used to inform the conversion of HGB to an ordinal value. We refer to this mix of binary and ordinal values as the *discretized* feature space. It is worth emphasizing that this score -- including the conventions used to convert the features to discrete space -- is part of the standard of care for patients with acute gastrointestinal bleeding, and has been extensively validated empirically (see Section 5 and Appendix E for additional details). \n\n**Summary of new results**\nWe attach the results of running ExpertTest in both the original and discretized feature space. In both cases, we further normalize each feature to lie in $[0, 1]$ (to ensure that no feature is given outsize weight in calculating pairwise similarity). We find strong evidence that physicians incorporate information other than the 9 features described above when making hospitalization decisions ($\\tau \\approx 0$). In the original feature space, which includes continuous features, we naturally fail to find identical pairs of patients. Nonetheless, the pairs we find are very close under the $\\ell_2$ norm, suggesting that ExpertTest remains approximately valid. In the discretized feature space we succeed in finding up to $L=1000$ pairs of identical patients, meaning that ExpertTest recovers (effectively) exact type I error control.\n\n**In response to**\n> In the case study, $\\hat{Y}$ is defined as whether the patient is admitted, and the ground truth $Y$ is defined as whether they suffered one of three adverse outcomes…It seemed like this might actually be a setup where the “prediction” and “ground truth” are not independent of each other.\"\n\n1. $Y$ and $\\hat{Y}$ will typically be correlated (as they are here, and in any setting where the human makes reasonable forecasts). The condition we require (see Section 6 and Appendix A) is that $\\hat{Y}$ does not *causally* influence $Y$. \n2. We make this assumption here because (1) the ER physician making hospitalization decisions and GI specialist making treatment decisions are different physicians working in a large hospital system -- indeed, there are many cases where the GI specialist immediately discharges a patient admitted by the ER physician -- and (2) we assume we can still observe $Y=1$ outcomes for discharged patients ($\\hat{Y} = 0$) who are readmitted or die within 30 days. \n3. This latter assumption is necessarily imperfect, but is consistent with standard 'adverse outcome' definitions in the literature and Center for Medicare Services (CMS) guidelines. Relevant citations and further discussion of this outcome definition are included in Section 5 and Appendix E. \n\n**In response to**\n> \"How sensitive is ExpertTest to the choice of loss function?\"\n\nIn the special case of binary outcomes and predictions (as in our case study), ExpertTest will provide identical results for any loss function which is strictly increasing in the number of prediction mistakes. This will be true of nearly any natural loss function. The results of ExpertTest also do not depend on the relative cost of false negatives and false positives, which may be arbitrarily different (e.g., false negatives in a clinical setting may be far more costly than false positives; we need not specify exactly *how much* more costly as our results are insensitive to this choice). **We discuss this phenomenon further in Appendix E.** In more general settings we also find that ExpertTest is also not very sensitive to the choice of loss function, and will include additional experiments to demonstrate this in our final submission (unfortunately we cannot do so here due to space constraints).\n\n"}, {"review_id": "f0opA8VVsW", "reviewer": "Reviewer_7Ctt", "summary": "This paper proposes a statistical framework for measuring whether, in the context of algorithmic predictions, human experts incorporate valuable information in their decision making that is unknowable to the algorithm. The authors formalize this question as a simple hypothesis test: “are human expert predictions independent from the outcome variable, when conditioned on the feature vector”.  The proposed statistical test for “presence of human expertise” is straightforward, drawing on prior literature on testing conditional independence, and provides interpretable p-values. The paper then uses this framework to analyze real-world medical data from an emergency department of a large hospital, showing that physicians do in fact incorporate information above and beyond that captured by a standard algorithmic screening tool.", "questions": "- In Section 2, first paragraph: why can you assume X is independent of U, without a loss of generality?\n- I would be interested in seeing this used in settings where algorithmic predictions are close to replacing human predictions. Radiology images are the immediate example that comes to mind.\n- Could your test (or a small modification) detect if human experts are using additional information in a negative way? For example, could you use it to detect biasedness in college admissions or medical treatment choices because the expert is relying on information outside of the set of features they should be looking at?", "limitations": "Yes, good discussion of limitations and prior work.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The main strengths of this paper are in it’s simplicity, clear exposition, and scoping of a well-defined problem that it tries to solve. The authors’ give an elegant framework for formally defining the problem of measuring valuable human expertise and their proposed test is quite intuitive. The paper presents all its ideas in a concise manner and also discusses its limitations quite candidly. \n\nI also like the thoroughness of the experiment conducted by the authors—it seems well executed and the results are compelling evidence for the validity of their statistical test.", "weaknesses": "Some weaknesses of this work:\n\n- From an algorithmic/technical standpoint, this paper uses a straightforward notion of conditional independence to define the problem, and a simple binned conditional independence test to solve it. This simplicity is not a bad thing, but it is worth noting that the main contribution of this work doesn’t present a novel technique or technical insight.\n- The results and experiments of this paper would likely not extend to a high-dimensional setting. Their current experiment uses discrete, scalar features. The authors discuss this in their limitations. A concrete example: how would a test like this work for, say, radiology images, where the human predictive distribution is likely not smooth w.r.t the $\\ell_2$ metric.\n- I understand it’s not an easy task, but this paper would be much stronger if there were additional experiments where this technique was employed to understand the interplay between human and algorithmic decision making.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback! Our responses are provided below.\n\n**In response to**\n>  \"could your test (or a small modification) detect if human experts are using additional information in a negative way?''\n\n1. Yes, this algorithm could absolutely be used in such a setting, though note that we require that outcomes $Y$ are not causally influenced by the predictions $\\hat{Y}$. This precludes the possibility that even observing the outcome is contingent on the human's decision, as is likely the case in e.g. college admissions. This is discussed in Section 6 and further in Appendix A. \n2. With that caveat, we could certainly test whether human forecasts are independent of sensitive attributes (e.g., race) after conditioning on some set of `allowable' features. This would be very similar to the definition of conditional statistical parity given in \"Algorithmic decision making and the cost of fairness'' (Corbett-Davies et. al, 2017). Alternatively, one could test whether the predictions $\\hat{Y}$ and sensitive attributes are independent conditional on the outcome $Y$, as in \"Equality of Opportunity in Supervised Learning'' (Hardt et. al, 2016).\n3. Depending on the quantity of interest -- for example, whether forecasts are *unrelated* to race or whether they are systematically *lower* for a particular race -- one might consider modifying our test to be a two-tailed rather than single-tailed hypothesis test. \n\n**In response to**\n> \"In Section 2, first paragraph: why can you assume X is independent of U, without a loss of generality?\"\n\nThis concern is addressed in our response to all reviewers (see above).\n\n**In response to**\n> \"The results and experiments of this paper would likely not extend to a high-dimensional setting.\"\n\nThis concern is addressed in our response to all reviewers (see above).\n\n**In response to**\n> \"I understand it’s not an easy task, but this paper would be much stronger if there were additional experiments where this technique was employed to understand the interplay between human and algorithmic decision making.\"\n\nThis concern is addressed in our response to all reviewers (see above)."}, {"review_id": "qArK1S4h1N", "reviewer": "Reviewer_NoBC", "summary": "While humans and machines oftentimes make differing decisions, it's unclear whether humans make these decisions based on extra factors or information unavailable to machines. To understand this situation, the paper proposes a statistical test to determine if expert predictions are independent of the labels, when accounting for the input features. This idea indicates whether humans rely on different information, and in a sense add additional value unseen by a model. To evaluate this, the paper analyzes doctor predictions in a hospital admitance system, and find that doctors tend to use additional information.", "questions": "1. What are other choices for the function F, and would they be equally valid? \n2. For the experiment with hospital admission data, would the results change if instead of using the GBS score, the individual factors that were used as inputs to the score were used? \n3. In table 2, what does mismatched pairs mean?\n", "limitations": "Authors discuss limitations fairly thoroughly.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Statistical approach is well motivated and clearly demonstrates how to test for humans relying on extra information\n2. Test allows for flexibility due to choice of L, allowing for different statistical properties\n3. Method is evaluated on a real-world hospital dataset, and the connection between the evaluation and the methodology is clear \n", "weaknesses": "1. Method relies on dataset containing pairs that are close in input space, yet distinct in feature space; such a situation might not be generalizable\n2. Evaluation is only done on one real-world dataset; a controlled evaluation of the test would give better insights into how the test performs and the impact of different parameters\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback! Our responses are provided below.\n\n**In response to**\n> \"What are other choices for the function F, and would they be equally valid?\"\n\nAs noted in Section 3, our test is valid for any choice of $F(\\cdot)$. However, the *power* of the test will certainly depend on the choice of $F(\\cdot)$. As is typical for hypothesis tests, the degree of this dependence is hard to characterize analytically since it depends on the specific distribution of $(X, Y, \\hat{Y})$. Our particular choice is a natural one however, as it is well-powered to detect the specific form of dependence we care about -- whether or not the unobservables $U$ actually improve human accuracy with respect to a known measure of accuracy. Please also see our response to 7cTX for additional discussion regarding the *choice* of loss function.\n\n**In response to**\n> \"In table 2, what does mismatched pairs mean?\"\n\n\"Mismatched pairs\" refers to the number of pairs of observations chosen by ExpertTest which are not exactly identical. In the context of our case study, it means that we chose a pair of patients who did not have identical Glasgow-Blatchford scores, which risks incurring additional type I error as described in Section 4. The definition of \"mismatched pairs\" is given in Section 5, but we will clarify this in our final draft by including a description of each column in the table caption.\n\n**In response to**\n> \"For the experiment with hospital admission data, would the results change if instead of using the GBS score, the individual factors that were used as inputs to the score were used?\"\n\nExcellent question -- please see our response to 7cTX for a detailed discussion of this point. The results of this experiment are included in the attached figures.\n   \n**In response to**\n> \"Evaluation is only done on one real-world dataset; a controlled evaluation of the test would give better insights into how the test performs and the impact of different parameters\"\n\nThis concern is addressed in our response to all reviewers (see above)."}]

Author rebuttal:
We thank all four of the reviewers for their thoughtful and constructive feedback. We address feedback provided by more than one reviewer in this comment, with additional responses to individual reviewer concerns provided in-line below each review.

# High dimensional feature spaces
> **7Ctt:** "The results and experiments of this paper would likely not extend to a high-dimensional setting. Their current experiment uses discrete, scalar features. The authors discuss this in their limitations. A concrete example: how would a test like this work for, say, radiology images, where the human predictive distribution is likely not smooth w.r.t the $\ell_2$ metric."

> **NoBC:** "Method relies on dataset containing pairs that are close in input space, yet distinct in feature space; such a situation might not be generalizable"


1. Per the proof of Theorem 2 (Appendix C), the excess type I error scales like $O(n^{-\frac{1}{d}})$ for a $d$ dimensional feature space. This behavior is typical in a generic, non-parametric setting (see e.g. the model-powered test of conditional independence of Sen et. al, 2017, which recovers the same bound).
2. If there were a *latent* lower dimensional representation, as is often the case with e.g., image data, our results could potentially be adapted to be with respect to the latent lower dimension. Indeed, as **7Ctt** points out, the smoothness condition required for Theorem 2 may not hold in a high dimensional setting, but may hold with respect to the latent representation.
3. It is also worth noting that these are *worst case* results over the set of possible distributions on $X, Y, \hat{Y}$. We find in both our real-world case study (Section 5 and figures attached to this rebuttal) and synthetic experiments (Appendix F) that typical distributions are better behaved; in particular, they exhibit more 'clustering' of similar $(x_i, x_j)$ pairs and thus require smaller datasets to achieve acceptable type I error. 
4. We discuss alternative tests (e.g., kernel based tests of conditional independence) which may be better suited for high-dimensional feature spaces in Section 6.


# Additional experiments, synthetic data
> **DMEm**: "Experimental validation: While it is great that the authors perform experiments on real-world data, it would be ideal to also verify the behavior of the test using synthetic where the differences between human and AI can be more carefully controlled to establish how sensitive the test is to these similarities and potentially the effect on the choice of L."

> **NoBC**: "Evaluation is only done on one real-world dataset; a controlled evaluation of the test would give better insights into how the test performs and the impact of different parameters"

1. Our manuscript includes extensive experiments on synthetic data in Appendix F, which is included with our original set of supplementary material. It is likely that the reviewers missed this section.
2. We reference these experiments in Section 1, and will feature them more prominently in our edited draft. 


# On the potential correlation of $X$ and $U$
> **DMEm:** "What would the test return if the human is using information U that is correlated with X? Here, this may be a case where $U$ is easier for a human to notice compared to $X$."

> **7Ctt:** "In Section 2, first paragraph: why can you assume X is independent of U, without a loss of generality?"

1. First, **there is indeed a typo in our work**; we intended to state that $X$ and $U$ can be assumed to be *uncorrelated* without loss of generality, not independent.
2. Neither assumption (independence or zero correlation) is used anywhere in our analysis however; this sentence was merely intended to clarify our framework.
3. Zero correlation can be assumed without loss of generality because we can take $U' = (U - E[U \mid X])$ and replace $U$ by $U'$. Thus, even if the human forecaster incorporates unobserved information which is correlated with $X$, we can define $U$ to be the *additional* information which is not correlated with the observed features. 
4. **Given the confusion around this point, our proposal is to simply delete this sentence from the final draft.** If the reviewers prefer, we are also happy to change 'independent' to 'uncorrelated' and clarify this point in the final draft.


# Formatting of tables
> **DMEm:** "The table captions are not descriptive, which hinders readability."

> **NoBC:** "In table 2, what does mismatched pairs mean?"

This is excellent feedback, and we will include more descriptive captions in the final draft. We answer NoBC's specific question in our individual response below.
 

Meta-review: The reviewers all agree on the contribution and novelty. Thus I recommend acceptance.
Decision: Accept (spotlight)