SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rohan Alur
ralur@mit.edu
Loren Laine
loren.laine@yale.edu
Darrick K. Li
darrick.li@yale.edu
Manish Raghavan
Sloan, EECS
mragh@mit.edu
Devavrat Shah
EECS, IDSS, LIDS, SDSC
devavrat@mit.edu
Dennis Shung
dennis.shung@yale.edu
High-stakes prediction tasks (e.g., patient diagnosis) are often handled by trained human experts. A common source of concern about automation in these settings is that experts may exercise intuition that is difficult to model and/or have access to information (e.g., conversations with a patient) that is simply unavailable to a would-be algorithm. This raises a natural question whether human experts add value which could not be captured by an algorithmic predictor.
We develop a statistical framework under which we can pose this question as a natural hypothesis test. Indeed, as our framework highlights, detecting human expertise is more subtle than simply comparing the accuracy of expert predictions to those made by a particular learning algorithm. Instead, we propose a simple procedure which tests whether expert predictions are statistically independent from the outcomes of interest after conditioning on the available inputs ('features'). A rejection of our test thus suggests that human experts may add value to any algorithm trained on the available data, and has direct implications for whether human-AI 'complementarity' is achievable in a given prediction task.
We highlight the utility of our procedure using admissions data collected from the emergency department of a large academic hospital system, where we show that physicians' admit/discharge decisions for patients with acute gastrointestinal bleeding (AGIB) appear to be incorporating information that is not available to a standard algorithmic screening tool. This is despite the fact that the screening tool is arguably more accurate than physicians' discretionary decisions, highlighting that - even absent normative concerns about accountability or interpretability - accuracy is insufficient to justify algorithmic automation.
Progress in machine learning, and in algorithmic decision aids more generally, has raised the prospect that algorithms may complement or even automate human decision making in a wide variety of settings. If implemented carefully, these tools have the potential to improve accuracy, fairness, interpretability and consistency in many prediction and decision tasks. However, a primary challenge in nearly all such settings is that some of the relevant inputs – ‘features,’ in machine learning parlance – are difficult or even impossible to encode in a way that an algorithm can easily consume. For example, doctors use direct conversations with patients to inform their diagnoses, and sports franchises employ professional scouts to qualitatively assess prospective players. One can think of these experts as incorporating information which is practically difficult to provide to an algorithm, particularly as
tabular data, or perhaps exercising judgment which is infeasible to replicate with a computational process. Either perspective presents a challenge when deploying predictive algorithmic tools, as any such model will necessarily fail to incorporate at least some of the information that a human expert might consider. Thus, as we seek to use algorithms to improve decision-making, we must answer the following question:
For a given prediction task, do human experts add value which could not be captured by any algorithmic forecasting rule?
The answer to this question has significant consequences: if experts are incorporating salient but hard-to-quantify information, we might attempt to somehow ensemble or combine the human and algorithmic predictions; this is commonly referred to as seeking 'complementarity' in the literature on human-machine interaction. On the other hand, if it appears that an expert is not extracting signal beyond whatever is contained in the available features, we might consider whether we can automate the prediction task entirely, or at least reduce the degree to which a human may override algorithmic recommendations.
At this stage it is worth asking – why not simply compare the prediction accuracy of a human expert to that of a particular predictive algorithm? If the human expert performs better than a competing algorithm, we might say the expert adds value which is not captured by the algorithm. However, as the example presented next illustrates, it is possible for the expert to incorporate information that could not be captured by any learning algorithm – even when the expert substantially underperforms a particular algorithm trained to accomplish the same task. Indeed, this is not just a hypothetical: a large body of prior literature (see e.g. Agrawal (2019) for a comprehensive overview) suggests that humans reliably underperform even simple statistical models, and in Section 5, we find exactly this dynamic in real-world patient triage data. Nonetheless, as we highlight next, humans may still add valuable information in a given forecasting task.
An illustration: experts may add information despite poor predictions. Let  $Y$  denote the outcome of interest and let  $X, U$  be features that drive the outcome. Specifically, let
$$
Y = X + U + \epsilon_ {1}, \tag {1}
$$
where  $\epsilon_{1}$  is some exogenous noise. For the purposes of this stylized example, we'll assume that  $X, U, \epsilon_{1}$  are all zero mean and pairwise independent random variables. Suppose the human expert can observe both  $X$  and  $U$ , but only  $X$  is made available to a predictive algorithm. An algorithm tasked with minimizing squared error might then seek to precisely estimate  $\mathbb{E}[Y|X] = X$ . In contrast, the expert may instead use simpler heuristics to construct an estimate which can be modeled as
$$
\hat {Y} = \operatorname {s i g n} (X) + \operatorname {s i g n} (U) + \epsilon_ {2}, \tag {2}
$$
where  $\epsilon_{2}$  is independent zero-mean noise, and can be thought of as modeling idiosyncracies in the expert's cognitive process<sup>1</sup>. As discussed in detail in Appendix F, there exist natural distributions over  $(X,U,\epsilon_1,\epsilon_2)$  such that the algorithm performs substantially better than the expert in terms of predictive accuracy. In fact, we show that there exist natural distributions where the algorithm outperforms the expert even under any linear post-processing of  $\hat{Y}$  (e.g., to correct for expert predictions which are highly correlated with  $Y$  but perhaps incorrectly centered or scaled). Nonetheless, the expert predictions clearly contain information (cf.  $\mathrm{sign}(U)$ ) that is not captured by the algorithm.
However, because  $U$  is not recorded in the available data, it is not obvious how to distinguish the above scenario from one in which the expert only extracts signal from  $X$ . For example, they might instead make predictions as follows:
$$
\hat {Y} = \operatorname {s i g n} (X) + \epsilon_ {2}. \tag {3}
$$
While a learning algorithm may outperform the expert in both cases, the expert in scenario (2) still captures valuable information; the expert in (3) clearly does not. The goal of this work will be to develop a test which allows us to distinguish between scenarios like these without the strong modeling assumptions made in this example.
<sup>1</sup>For example, a well-known study by Eren and Mocan (2018) demonstrates that unexpected losses by the Louisiana State University football team lead judges to hand out longer juvenile sentences; this is a form of capricious decision making which will manifest as noise  $(\epsilon_{2})$  in an analysis of sentencing decisions.
Contributions. To understand whether human experts can add value for a given prediction task, we develop a statistical framework under which answering this question becomes a natural hypothesis test. We then provide a simple, data-driven procedure to test this hypothesis. Our proposed algorithm takes the form of a conditional independence test, and is inspired by the Model-X Knockoffs framework of Candès et al. (2016), the Conditional Permutation Test of Berrett et al. (2018) and the 'Model-Powered' test of Sen et al. (2017). Our test is straightforward to implement and provides transparent, interpretable p-values.
Our work is closely related to a large body of literature comparing human performance to that of an algorithm (Cowgill (2018), Dawes et al. (1989), Grove et al. (2000), among others), and developing learning algorithms which are complementary to human expertise (Madras et al. (2018), Raghu et al. (2019), Mozannar and Sontag (2020), Keswani et al. (2021), Agrawal et al. (2018), Bansal et al. (2020), Rastogi et al. (2022) and Bastani et al. (2021)). However, although similarly motivated, we address a different problem which is in a sense 'upstream' of these works, as we are interested in testing for whether a human forecaster demonstrates expertise which cannot be replicated by any algorithm. Thus, we think of our test as assessing as a necessary condition for achieving human-AI complementarity; success in practice will further depend on the ability of a mechanism designer to actually incorporate human expertise into some particular algorithmic pipeline or feedback system. We discuss these connections further in Appendix A.
We apply our test to evaluate whether emergency room physicians incorporate valuable information which is not available to a common algorithmic risk score for patients with acute gastrointestinal bleeding (AGIB). To that end, we utilize patient admissions data collected from the emergency department of a large academic hospital system. Consistent with prior literature, we find that this algorithmic score is an exceptionally sensitive measure of patient risk – and one that is highly competitive with physicians' expert assessments. Nonetheless, our test provides strong evidence that physician decisions to either hospitalize or discharge patients with AGIB are incorporating valuable information that is not captured by the screening tool. Our results highlight that prediction accuracy is not sufficient to justify automation of a given prediction task. Instead, our results make a case for experts working with a predictive algorithm, even when algorithms might handily outperform their human counterparts.
Organization. In Section 2, we formalize the problem of auditing for expertise, and in Section 3 we present our data-driven hypothesis test. Section 4 then examines the theoretical properties of the test. In Section 5 we present our empirical findings from applying this test to real-world patient triage data. Finally, Section 6 provides discussion of our results and directions for future work. We also include a discussion of additional related work in Appendix A, and provide numerical simulations to corroborate our theoretical and empirical results in Appendix F $^2$ .
2 Setup and question of interest
We consider a generic prediction task in which the goal is to forecast some outcome  $Y \in \mathbb{R}$  on the basis of observable features  $X \in \mathcal{X}$ . The human expert may additionally have access to some auxiliary private information  $U \in \mathcal{U}$ . For concreteness, let  $\mathcal{X} = \mathbb{R}^d$  for some  $d \geq 1$ .
We posit that the outcome  $Y$  is generated as follows: for some unknown function  $f:\mathcal{X}\times \mathcal{U}\to \mathbb{R}$
$$
Y = f (X, U) + \epsilon_ {1}, \tag {4}
$$
where, without loss of generality,  $\epsilon_{1}$  represents mean zero idiosyncratic noise with unknown variance.
We are also given predictions by a human expert, denoted as  $\hat{Y}$ . We posit that the expert predictions  $\hat{Y}$  are generated as follows: for some unknown function  $\hat{f}:\mathcal{X}\times \mathcal{U}\to \mathbb{R}$ ,
$$
\hat {Y} = \hat {f} (X, U) + \epsilon_ {2}, \tag {5}
$$
where  $\epsilon_{2}$  also captures mean zero idiosyncratic noise with unknown variance.
We observe  $(X,Y,\hat{Y})$  which obey (4)-(5); the private auxiliary feature  $U$  is not observed. Concretely, we observe  $n$  data points  $(x_{i},y_{i},\hat{y}_{i}), i\in [n]\equiv \{1,\dots ,n\}$
$^{2}$ Code, data and instructions to replicate our experiments are available at https://github.com/ralur/auditing-human-expertise. Publication of the results and data associated with the empirical study in section 5 have been approved by the relevant institutional review board (IRB).
Our goal is to answer the question "do human experts add information which could not be captured by any algorithm for a given prediction task?" We assume that any competing learning algorithm can utilize  $X$  to predict  $Y$ , but it can not utilize  $U$ . Thus, our problem reduces to testing whether  $U$  has some effect on both  $Y$  and  $\hat{Y}$ , which corresponds to the ability of an expert to extract signal about  $Y$  from  $U$ . If instead  $U$  has no effect on  $Y$ ,  $\hat{Y}$  or both (either because  $U$  is uninformative about  $Y$  or the expert is unable to perceive this effect), then conditioned on  $X$ ,  $Y$  and  $\hat{Y}$  are independent. That is, if the human expert fails to add any information which could not be extracted from the observable features  $X$ , the following must hold:
$$
H _ {0}: Y \perp \hat {Y} | X. \tag {6}
$$
Intuitively,  $H_0$  captures the fact that once we observe  $X$ ,  $\hat{Y}$  provides no additional information about  $Y$  unless the expert is also making use of some unobserved signal  $U$  (whether explicitly or implicitly). In contrast, the rejection of  $H_0$  should be taken as an evidence that the expert (or experts) can add value to any learning algorithm trained on the observable features  $X \in \mathcal{X}$ ; indeed, a strength of this framework is that it does not require specifying a particular algorithmic baseline. However, it's worth remarking that an important special case is to take  $X$  to be the prediction made by some specific learning algorithm trained to forecast  $Y$ . In this setting, our test then reduces to assessing whether  $\hat{Y}$  adds information to the predictions made by this learning algorithm, and can be viewed as a form of feature selection.
Goal. Test the null hypothesis  $H_0$  using observed data  $(x_i, y_i, \hat{y}_i), i \in [n] \equiv \{1, \dots, n\}$ .
To make this model concrete, in Section 5 we use our framework to test whether emergency room physicians incorporate information that is not available to a common algorithmic risk score when deciding whether to hospitalize patients. Accordingly, we let  $X \in [0,1]^{9}$  be the inputs to the risk score,  $\hat{Y} \in \{0,1\}$  be a binary variable indicating whether a given patient was hospitalized, and  $Y \in \{0,1\}$  be an indicator for whether, in retrospect, a patient should have been hospitalized. The risk score alone turns out to be a highly accurate predictor of  $Y$ , but physicians take many other factors into account when making hospitalization decisions. We thus seek to test whether physicians indeed extract signal which is not available to the risk score  $(Y \not\perp \hat{Y} \mid X)$ , or whether attempts to incorporate other information and/or exercise expert judgement simply manifest as noise  $(Y \perp \hat{Y} \mid X)$ .
3 ExpertTest: a statistical test for human expertise
To derive a statistical test of  $H_0$ , we will make use of the following elementary but powerful fact about exchangeable random variables.
A test for exchangeability. Consider  $K + 1$  random variables  $(Z_0, \ldots, Z_K)$  which are exchangeable, i.e. the joint distribution of  $(Z_0, \ldots, Z_K)$  is identical to that of  $(Z_{\sigma(0)}, \ldots, Z_{\sigma(K)})$  for any permutation  $\sigma : \{0, \ldots, K\} \to \{0, \ldots, K\}$ . For example, if  $Z_0, \ldots, Z_K$  are independent and identically distributed (i.i.d.), then they are exchangeable. Let  $F$  be a function that maps these variables to a real value. For any such  $F(\cdot)$ , it can be verified that the order statistics (with any ties broken uniformly at random) of  $F(Z_0), \ldots, F(Z_K)$  are uniformly distributed over  $(K + 1)!$  permutations of  $\{0, \ldots, K\}$ . That is,  $\tau_K$  defined next, is distributed uniformly over  $\{0, 1/K, 2/K, \ldots, 1\}$ :
$$
\tau_ {K} = \frac {1}{K} \sum_ {k = 1} ^ {K} \mathbb {1} [ F (Z _ {0}) \lesssim F (Z _ {k}) ] \tag {7}
$$
where we use definition  $\mathbb{1}[\alpha \lesssim \beta] = 1$  if  $\alpha < \beta$  and  $0$  if  $\alpha > \beta$ . If instead  $\alpha = \beta$ , we independently assign it to be  $1$  or  $0$  with equal probability. Thus, if  $(Z_0, \ldots, Z_K)$  are exchangeable, then  $\mathbb{P}(\tau_K \leq \alpha) \leq \alpha + 1 / (K + 1)^{K \to \infty} \alpha$  and we can reject the hypothesis that  $(Z_0, \ldots, Z_K)$  are exchangeable with p-value (effectively) equal to  $\tau_K$ .
Observe that while this validity guarantee holds for any choice of  $F(\cdot)$ , the power of the test will depend crucially on this choice; for example, a constant function which maps every argument to the same value would have no power to reject the null hypothesis. We return to the choice of  $F(\cdot)$  below.
Constructing exchangeable distributions. We will leverage the prior fact about the order statistics of exchangeable random variables to design a test of  $H_0: Y \perp \hat{Y} \mid X$ . In particular, we would like to use the observed data to construct  $K + 1$  random variables that are exchangeable under  $H_0$ , but not exchangeable otherwise. To that end, consider a simplified setting where  $n = 2$ , with  $x_1 = x_2 = x$ . Thus, our observations are  $Z_0 = \{(x, y_1, \hat{y}_1), (x, y_2, \hat{y}_2)\}$ . Suppose we now sample  $(\tilde{y}_1, \tilde{y}_2)$  uniformly at random from  $\{(\hat{y}_1, \hat{y}_2), (\hat{y}_2, \hat{y}_1)\}$ . That is, we swap the observed values  $(\hat{y}_1, \hat{y}_2)$  with probability  $\frac{1}{2}$  to construct a new dataset  $Z_1 = \{(x, y_1, \tilde{y}_1), (x, y_2, \tilde{y}_2)\}$ .
Under  $H_0$ , it is straightforward to show that  $Z_0$  and  $Z_1$  are independent and identically distributed conditioned on observing  $(x, x)$ ,  $(y_1, y_2)$  and either  $(\hat{y}_1, \hat{y}_2)$  or  $(\hat{y}_2, \hat{y}_1)$ . That is,  $Z_0, Z_1$  are exchangeable, which will allow us to utilize the test described above for  $H_0$ .
Why condition on this somewhat complicated event? Intuitively, we would like to resample  $\tilde{Y} = (\tilde{y}_1,\tilde{y}_2)$  from the distribution of  $\mathcal{D}_{\hat{Y} |X}$ ; under the null,  $(x,y,\hat{y})$  and  $(x,y,\tilde{y})$  will be exchangeable by definition. However, this requires that we know (or can accurately estimate) the distribution of  $\hat{Y} \mid X$ , which in turn requires modeling the expert's decision making directly. Instead, we simplify the resampling process by only considering a swap of the observed  $\hat{y}$  values between identical values of  $x$  - this guarantees exchangeable data without modeling  $\mathcal{D}_{\hat{Y} |X}$  at all!
This approach can be extended for  $n$  larger than 2. Specifically, if there are  $L$  pairs of identical  $x$  values, i.e.  $x_{2\ell -1} = x_{2\ell}$  for  $1 \leq \ell \leq L$ , then it is possible to construct i.i.d.  $Z_0,\ldots ,Z_K$  for larger  $K$  by randomly exchanging values of  $\hat{y}$  for each pair of data points.
As discussed above, we'll also need to choose a particular function  $F(\cdot)$  to apply to  $Z_{0}$  and  $Z_{1}$ . A natural, discriminatory choice of  $F$  is a loss function: for example, given  $D = \{(x_{i},y_{i},\hat{y}_{i}):i\leq 2L\}$ , let  $F(D) = \sum_{i}(y_{i} - \hat{y}_{i})^{2}$ . This endows  $\tau_{K}$  with a natural interpretation – it is the probability that an expert could have performed as well as they did (with respect to the chosen loss function  $F$ ) by pure chance, without systematically leveraging some unobserved  $U$ .
Of course, in practice we are unlikely to observe many pairs where  $x_{2\ell - 1} = x_{2\ell}$ , particularly when  $x$  takes value in a non-finite domain, e.g. [0, 1] or  $\mathbb{R}$ . However, if the conditional distribution of  $\hat{Y} | X$  is nearly the same for close enough values of  $X = x$  and  $X = x'$ , then we can use a similar approach with some additional approximation error. This is precisely the test that we describe next.
ExpertTest. Let  $L \geq 1$  be an algorithmic parameter and  $m: \mathcal{X} \times \mathcal{X} \to \mathbb{R}_{\geq 0}$  be some distance metric over  $\mathcal{X}$ , e.g. the  $\ell_2$  distance. Let  $F(\cdot)$  be some loss function of interest, e.g. the mean squared error.
First, compute  $m(x_{i},x_{j}):i\neq j\in [n]$  and greedily select  $L$  disjoint pairs which are as close as possible under  $m(\cdot ,\cdot)$ . Denote these pairs by  $\{(x_{i_{2\ell -1}},x_{i_{2\ell}}):\ell \in [L]\}$ .
Let  $D_0 = \{(x_i, y_i, \hat{y}_i) : i \in \{i_{2\ell-1}, i_{2\ell} : \ell \in [L]\}\}$  denote the observed dataset restricted to the  $L$  chosen pairs. Let  $D_1$  be an additional dataset generated by independently swapping each pair  $(\hat{y}_{i_{2\ell-1}}, \hat{y}_{i_{2\ell}})$  with probability  $1/2$ , and repeat this resampling procedure to generate  $D_1 \ldots D_K$ . Next, compute  $\tau_K$  as follows:
$$
\tau_ {K} = \frac {1}{K} \sum_ {k = 1} ^ {K} \mathbb {1} [ F (D _ {0}) \lesssim F (D _ {k}) ] \tag {8}
$$
Finally, we reject the hypothesis  $H_0$  with  $p$ -value  $\alpha + 1 / (K + 1)$  if  $\tau_K \leq \alpha$  for any desired confidence level  $\alpha \in (0,1)$ . Our test is thus quite simple: find  $L$  pairs of points that are close under some distance metric  $m(\cdot, \cdot)$ , and create  $K$  synthetic datasets by swapping the expert forecasts for each pair independently with probability  $1/2$ . If the expert's loss on the original dataset is "small" relative to the loss on these resampled datasets, this is evidence that the synthetic datasets are not exchangeable with the original, and thus, the expert is using some private information  $U$ .
Of course, unlike in the example above, we swapped pairs of predictions for different values of  $x$ . Thus,  $D_0 \ldots D_K$  are not exchangeable under  $H_0$ . However, we'll argue that because we paired "nearby" values of  $x$ , these datasets are "nearly" exchangeable. These are the results we present next.
We provide theoretical guarantees associated with the ExpertTest. First, we demonstrate the validity of our test in a generic setting. That is, if  $H_0$  is true, then ExpertTest will not reject it with high probability. We then quantify this guarantee precisely under a meaningful generative model.
To state the validity result, we need some notation. For any  $(x,\hat{y})$  and  $(x^{\prime},\hat{y}^{\prime})$ , define the odds ratio
$$
r \left(\left(x, \hat {y}\right), \left(x ^ {\prime}, \hat {y} ^ {\prime}\right)\right) = \frac {Q \left(\hat {Y} = \hat {y} \mid X = x\right) \times Q \left(\hat {Y} = \hat {y} ^ {\prime} \mid X = x ^ {\prime}\right)}{Q \left(\hat {Y} = \hat {y} ^ {\prime} \mid X = x\right) \times Q \left(\hat {Y} = \hat {y} \mid X = x ^ {\prime}\right)}, \tag {9}
$$
where  $Q(\cdot |\cdot)$  represents the density of the conditional distribution of human predictions  $\hat{Y} |X$  under  $H_0$ . For simplicity, we assume that such a conditional density exists.
Theorem 1 (Validity of ExpertTest) Given  $\alpha \in (0,1)$  and parameters  $K\geq 1,L\geq 1$  , the Type I error of ExpertTest satisfies
$$
\mathbb {P} \left(\tau_ {K} \leq \alpha\right) \leq \alpha + \left(1 - \left(1 - \varepsilon_ {n, L} ^ {*}\right) ^ {L}\right) + \frac {1}{K + 1}. \tag {10}
$$
Where  $\varepsilon_{n,L}^{*}$  is defined as follows
$$
\varepsilon_ {n, L} ^ {*} = \max  _ {\ell \in [ L ]} \left| \frac {1}{1 + r \left(\left(x _ {i _ {2 \ell - 1}} , \hat {y} _ {i _ {2 \ell - 1}}\right) , \left(x _ {i _ {2 \ell}} , \hat {y} _ {i _ {2 \ell}}\right)\right)} - \frac {1}{2} \right|. \tag {11}
$$