SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ben Chugg<sup>1</sup>, Santiago Cortes-Gomez<sup>1</sup>, Bryan Wilder<sup>1</sup>, Aaditya Ramdas<sup>1,2</sup>
Departments of Machine Learning<sup>1</sup> and Statistics<sup>2</sup>
{benchugg, scortesg, bwilder, aramdas}@cmu.edu
We provide practical, efficient, and nonparametric methods for auditing the fairness of deployed classification and regression models. Whereas previous work relies on a fixed-sample size, our methods are sequential and allow for the continuous monitoring of incoming data, making them highly amenable to tracking the fairness of real-world systems. We also allow the data to be collected by a probabilistic policy as opposed to sampled uniformly from the population. This enables auditing to be conducted on data gathered for another purpose. Moreover, this policy may change over time and different policies may be used on different subpopulations. Finally, our methods can handle distribution shift resulting from either changes to the model or changes in the underlying population. Our approach is based on recent progress in anytime-valid inference and game-theoretic statistics—the "testing by betting" framework in particular. These connections ensure that our methods are interpretable, fast, and easy to implement. We demonstrate the efficacy of our approach on three benchmark fairness datasets.
As algorithmic decision-making continues to increase in prevalence across both the private and public sectors [1, 2], there has been an increasing push to scrutinize the fairness of these systems. This has lead to an explosion of interest in so-called "algorithmic fairness", and a significant body of work has focused on both defining fairness and training models in fair ways (e.g., [3-5]). However, preventing and redressing harms in real systems also requires the ability to audit models in order to assess their impact; such algorithmic audits are an increasing area of focus for researchers and practitioners [6-9]. Auditing may begin during model development [10], but as model behavior often changes over time throughout real-world deployment in response to distribution shift or model updates [11, 12], it is often necessary to repeatedly audit the performance of algorithmic systems over time [8, 13]. Detecting whether deployed models continue to meet various fairness criteria is of paramount importance to deciding whether an automated decision-making system continues to act reliably and whether intervention is necessary.
In this work, we consider the perspective of an auditor or auditing agency tasked with determining if a model deployed "in the wild" is fair or not. Data concerning the system's decisions are gathered over time (perhaps with the purpose of testing fairness but perhaps for another purpose) and our goal is to determine if there is sufficient evidence to conclude that the system is unfair. If a system is in fact unfair, we want to determine so as early as possible, both in order to avert harms to users and because auditing may require expensive investment to collect or label samples [8, 13]. Following the recent work of Taskesen et al. [14] and Si et al. [15], a natural statistical framework for thinking about this problem is hypothesis testing. Informally, consider the null and alternative hypotheses,  $H_0$  and  $H_1$ , defined respectively as
$H_0$  : the model is fair,  $H_{1}$  : the model is unfair.
Unfortunately, traditional hypothesis testing requires stringent assumptions on the data; a fixed number of iid data points, for example. Such assumptions are unrealistic in our setting. We should, for instance, be able to continually test a system as we receive more information, i.e., perform sequential hypothesis testing. Moreover, we would like to be able stop collecting additional samples at arbitrary data-dependent stopping times if we have sufficient evidence against the null. This is not allowed in traditional statistical frameworks, where it is known as "peeking" or "p-hacking."
To overcome these challenges, we take advantage of recent progress in safe, anytime-valid inference (SAVI) to construct sequential procedures for determining whether an existing decision-making system is fair. SAVI is part of sequential analysis, a branch of statistics concerned—as the name suggests—with analyzing data sequentially while maintaining statistical validity. This subfield traces its origins back to Wald, Lai, Robbins, and several others beginning in the 1940s [16-20]. More recently, the power of sequential analysis to enable inference under continuous monitoring of data and data-dependent stopping rules (hence safe and anytime-valid) has led to an explosion of work in the area [21-23]. A further exciting development has been the connection between such methods and so-called "game-theoretic probability" [24, 25] which is consistent with modern measure theoretic probability<sup>1</sup> but provides an alternative foundation based on repeated games. Importantly for our purposes, this renders many of the tools of SAVI interpretable as well as statistically powerful. We refer the interested reader to the recent survey by Ramdas et al. [27] for further detail on the relationship between sequential analysis, SAVI, and game-theoretic statistics.
Contributions. We develop tools to sequentially audit both classifiers and regressors. In particular:
1. We formulate the problem of auditing the fairness of classification and regression models in terms of sequential hypothesis testing. The focus on sequential testing is distinct from other work, and highlights various desiderata that are important in practice—specifically, (i) being able to continuously monitor the data and (ii) a focus on rejecting the null as early as possible (i.e., reducing the number of costly samples needed to detect unfairness).
2. Next, we design nonparametric sequential hypothesis tests which hold under various definitions of group fairness. We treat auditing as sequential two-sample testing and adapt the recent work of Shekhar and Ramdas [28] on two-sample testing by betting to the fairness setting. We also provide novel bounds on the expected stopping time of our tests under the alternative, and demonstrate how to handle distribution drift (due to either model changes or changes in the underlying population), time-varying data collection policies, and composite nulls which more accurately reflect practical demands.
3. Finally, we demonstrate the real world applicability of our methods on three datasets: credit default data, US census data, and insurance data. We show that our method is robust to distribution shift resulting from model retraining and to various randomized data-collection policies whose densities deviate significantly from the underlying population. All code is publicly available at https://github.com/bchugg/auditing-fairness.
In order to provide a preview of our methodology, we suggest the following thought experiment. Imagine a fictitious better who is skeptical that a machine learning system is fair. She sets up an iterated game wherein she bets on the results of the audit as it is conducted. Her bets are structured such that, if the system is unfair, her expected payoff will be large. Conversely, if the system is fair, her expected payoff is small. Thus, if her wealth increases over time, it is evidence that the system is unfair. Our sequential test thus rejects the null if her wealth crosses some predetermined threshold. Of course, the trick is to design her bets in such a way the above conditions are satisfied, and that under the alternative, her wealth grows as quickly as possible. Mathematically, the above is made rigorous via nonnegative (super)martingales and results concerning their behavior over time [29, 30].
Additional Related Work. This work sits at the intersection of several fields. On the fairness side, the closest work to ours is that of Taskesen et al. [14] and Si et al. [15], both of which study fairness through the lens of (fixed-time) hypothesis testing, based on a batch of  $n$  iid observations. Given iid data  $Z_{t} \sim \rho$ , they formulate the null as  $H_0: \rho \in \mathcal{G}$ , where  $\mathcal{G}$  is the set of all "fair" distributions and derive a test statistic based on projecting the empirical distribution onto  $\mathcal{G}$  (minimizing a Wasserstein distance). They work with classifiers only. On the more technical side, we view our work as
There are some subtleties to this remark, but they are outside the scope of this work. However, as an example of how the insights of game-theoretic statistics can be applied to modern probability, we highlight the recent work of Waudby-Smith and Ramdas [26] which draws inspiration from Shafer and Vovk [25].
a continuation and application of the testing by betting framework [21] and more specifically of nonparametric two sample testing by betting [28]. We will deploy similar "betting strategies" (to be defined later) as [28], but provide novel analyses which are particular to our setting, and consider several extensions. Relatedly, but slightly further removed from our setting, Duan et al. [31] also employ game-theoretic ideas to construct interactive rank tests. The idea of a auditing a system to verify its veracity extends well beyond the domain of algorithmic fairness. Statistical procedures have been developed, for instance, to audit election results [32] and recently several SAVI inspired methods have been developed for several such scenarios [33, 34]. More broadly and outside the scope of this paper, betting techniques have been deployed with great success in optimization, online learning, and statistics [35-38, 26].
Consider a feature space  $\mathcal{X}$  and model  $\varphi : \mathcal{X} \to [0,1]$ . Depending on the application,  $\varphi(x)$  might be a risk score for an individual with covariates  $x$ , a classification decision, or the probability that some action is taken. Each  $x \in \mathcal{X}$  is associated with some "sensitive attribute"  $A = A_x \in \{0,1,\dots,J\}$  (indicating, e.g., the result of a private health test, whether they went to college, their income strata, etc). We will often refer to  $A$  as group membership. The classifier may or may not observe  $A$ . For the sake of exposition, we will assume that  $A \in \{0,1\}$ , i.e., there are two groups. However, our methods extend straightforwardly to more than two groups. The details are provided in Appendix B. Before formally stating the problem, let us discuss the notion of fairness we will employ in this work.
Fairness. We focus on the concept of "group" fairness. Roughly speaking, this involves ensuring that groups of individuals sharing various attributes are treated similarly. There have been multiple notions of group fairness proposed in the literature. We introduce the following generalized notion of group fairness, which can be instantiated to recapture various others.
Definition 1. Let  $\{\xi_j(A,X,Y)\}_{j\in \{0,\dots,J\}}$  denote a family of conditions on the attributes  $A$ , covariates  $X$ , and outcomes  $Y$ . We say a predictive model  $\varphi :\mathcal{X}\to [0,1]$  is fair with respect to  $\{\xi_j\}$  and a distribution  $\rho$  over  $\mathcal{X}$  if, for all  $i,j\in [J]$ ,  $\mathbb{E}_{X\sim \rho}[\varphi (X)|\xi_i(A,X,Y)] = \mathbb{E}_{X\sim \rho}[\varphi (X)|\xi_j(A,X,Y)]$ .
As was mentioned above, unless otherwise stated we will assume that there are only two conditions  $\xi_0,\xi_1$ . For our purposes, the important feature of Definition 1 is that it posits the equality of means. Indeed, letting  $\mu_{b} = \mathbb{E}_{X\sim \rho}[\varphi (X)|\xi_{b}]$  for  $b\in \{0,1\}$  we can write our null as  $H_0:\mu_0 = \mu_1$  and alternative as  $H_{1}:\mu_{0}\neq \mu_{1}$ . Different choices of conditions  $\xi_{j}$  lead to various fairness notions in the literature. For instance, if  $\varphi$  is a classification model, then:
1. Taking  $\xi_0 = \{A = 0, Y = 1\}$  and  $\xi_1 = \{A = 1, Y = 1\}$  results in equality of opportunity [39]. Predictive equality [40] is similar, corresponding to switching  $Y = 1$  with  $Y = 0$ .
2. Taking  $\xi_0 = \{A = 0\}$ ,  $\xi_1 = \{A = 1\}$  results in statistical parity [41].
3. Taking  $\xi_{j} = \{A = j,\ell (X)\}$  for some projection mapping  $\ell :\mathcal{X}\to F\subset \mathcal{X}$  onto "legitimate factors" results in conditional statistical parity [42, 41].
Problem Formulation. We consider an auditor who is receiving two streams of predictions  $Z^0 = (\varphi(X_t^0))_{t \in T_0}$  and  $Z^1 = (\varphi(X_t^1))_{t \in T_1}$ , where  $X_t^b$  obeys condition  $\xi_b$  (i.e., is drawn from some distribution over  $\mathcal{X}|\xi_b$ ). For brevity, we will let  $\widehat{Y}_t^b = \varphi(X_t^b)$ ,  $b \in \{0,1\}$ . The index sets  $T_0, T_1 \subset \mathbb{N} \cup \{\infty\}$  denote the times at which the predictions are received. We let the index differ between groups as it may not be feasible to receive a prediction from each group each timestep. We refer to this entire process as an audit of the model  $\varphi$ .
For  $b \in \{0,1\}$ , let  $T_{b}[t] = T_{b} \cap [t]$  be the set of times at which we receive predictions from group  $b$  up until time  $t$ . The auditor is tasked with constructing a sequential hypothesis test  $\phi \equiv (\phi_{t})_{t \geq 1}$  where  $\phi_{t} = \phi_{t}\big((\cup_{t \in T_{0}[t]} Z_{t}^{0}) \cup (\cup_{t \in T_{1}[t]} Z_{t}^{1})\big) \in \{0,1\}$  is a function of all predictions received until time  $t$ . We interpret  $\phi_{t} = 1$  as "reject  $H_{0}$ ", and  $\phi_{t} = 0$  as "fail to reject  $H_{0}$ ". Once we reject the null, we stop gathering data. That is, our stopping time is  $\tau = \arg \inf_{t} \{\phi_{t} = 1\}$ . We say that  $\phi$  is a level- $\alpha$  sequential test if
$$
\sup  _ {P \in H _ {0}} P (\exists t \geq 1: \phi_ {t} = 1) \leq \alpha , \text {o r e q u i v a l e n t l y} \sup  _ {P \in H _ {0}} P (\tau <   \infty) \leq \alpha . \tag {1}
$$
In words, the test should have small false positive rate (type I error) simultaneously across all time steps. (Note that a fixed-time level-  $\alpha$  test would simply drop the quantifier  $\exists t\geq 1$  and concern
itself with some fixed time  $t = n$ .) We also wish to design tests with high power. Formally, we say that  $\phi$  has asymptotic power  $1 - \beta$  if  $\sup_{P \in H_1} P(\forall t \geq 1 : \phi_t = 0) \leq \beta$ , or equivalently  $\sup_{P \in H_1} P(\tau = \infty) \leq \beta$ . That is, in all worlds in which the alternative is true, we fail to reject with probability at most  $\beta$  (type II error). Typically the type-II error  $\beta \equiv \beta_n$  decreases with the sample size  $n$ . In this work we will develop asymptotic power one tests, meaning that  $\beta_n \to 0$  as  $n \to \infty$ .
Martingales and filtrations. Our techniques rely on the machinery of nonnegative (super)martingales and results concerning their behavior over time. We introduce some of the technicalities here. A (forward) filtration  $\mathcal{F} \equiv (\mathcal{F}_t)_{t \geq 0}$  is an increasing sequence of  $\sigma$ -fields  $\mathcal{F}_t \subset \mathcal{F}_{t+1}$ . Throughout this paper, we will consider the "canonical filtration"  $\mathcal{F}_t = \sigma(Z_1, \ldots, Z_t)$  which can heuristically (but very usefully) be thought of as all the information known at time  $t$ . We say that a stochastic process  $S = (S_t)_{t \geq 1}$  is adapted to  $(\mathcal{F}_t)$  if  $S_t$  is  $\mathcal{F}_t$  measurable for all  $t \geq 1$ , and predictable if  $S_t$  is  $\mathcal{F}_{t-1}$  measurable for all  $t \geq 1$ . A  $P$ -martingale is an adapted stochastic process  $M = (M_t)_{t \geq 1}$  such that  $\mathbb{E}_P[M_{t+1} | \mathcal{F}_t] = M_t$  for all  $t \geq 1$ . If the equality is replaced with  $\leq$ , then  $M$  is a  $P$ -supermartingale. A particularly useful result in sequential analysis is Ville's inequality [29], which states that if  $M$  is a nonnegative  $P$ -supermartingale, then for all  $\alpha > 0$ ,  $P(\exists t \geq 0 : M_t \geq 1 / \alpha) \leq \alpha \mathbb{E}_P[M_0]$ . We will also employ the following randomized improvement to Ville's inequality [43, Corollary 4.1.1]: For  $M$  as above and any  $\mathcal{F}$ -stopping time  $\tau$ ,  $P(\exists t \leq \tau : M_t \geq 1 / \alpha$  or  $M_\tau \geq U / \alpha) \leq \alpha$ , where  $U$  is a uniform random variable on  $[0, 1]$  which is independent of  $M$  and  $\tau$ .
We begin by introducing the methodology in the setting where the predictions are received uniformly at random from the population. We will then progressively generalize the setting: Sections 3.2 and 3.3 will enable time-varying data collection policies, Section 3.4 will allow the means  $\mu_0$  and  $\mu_{1}$  to change with time, and Section 3.5 will consider composite nulls of the form  $|\mu_0 - \mu_1| \leq \epsilon$ .
To ease the presentation, let us make the assumption that we receive an audit from both groups each timestep, so that  $T_0 = T_1 = \mathbb{N}$ . This is without loss of generality; one can simply wait until multiple audits from each group are available. However, Appendix A provides a more detailed discussion on how to modify our framework and theorems if this condition is unmet.
3.1 Testing by betting
Returning to the intuition for a moment, recall the fictitious better discussed in the introduction. She attempts to prove that the system is unfair by betting on the results of the audits before they are revealed. If the system is unfair, then the average difference between  $\widehat{Y}_t^0$  and  $\widehat{Y}_t^1$  (i.e., the model's outputs across different groups) will be non-zero. The skeptic's bets are therefore structured such that if  $\mu_0 = \mathbb{E}\widehat{Y}_t^0\neq \mathbb{E}\widehat{Y}_t^1 = \mu_1$ , then her expected wealth will increase over time. More formally, at time  $t$ , the skeptic designs a payoff function  $S_{t}:\mathbb{R}\times \mathbb{R}\to \mathbb{R}_{\geq 0}$  which is  $\mathcal{F}_{t - 1}$  measurable and  $\mathbb{E}_P[S_t(\widehat{Y}_t^0,\widehat{Y}_t^1)|\mathcal{F}_{t - 1}]\leq 1$  if  $P\in H_0$  (i.e.,  $\mu_0 = \mu_1$ ). Next, the skeptic receives the model predictions  $\widehat{Y}_t^0 = \varphi (X_t^0)$  and  $\widehat{Y}_t^1 = \varphi (X_t^1)$ . We assume the skeptic starts with wealth of  $\mathcal{K}_0 = 1$ . At each time  $t$ , she reinvests all her wealth  $\kappa_{t - 1}$  on the outcome and her payoff is  $\mathcal{K}_t = S_t(\widehat{Y}_t^0,\widehat{Y}_t^1)\cdot \mathcal{K}_{t - 1} = \prod_{i = 1}^{t}S_{t}(\widehat{Y}_{t}^{0},\widehat{Y}_{t}^{1})$ . We call  $(\mathcal{K}_t)_{t\geq 0}$  the skeptic's wealth process. The wealth process is a supermartingale starting at 1 under the null, due to the constraint on the payoff function. Thus, by Ville's inequality, the probability that  $\kappa_{t}$  ever exceeds  $1 / \alpha$  is at most  $\alpha$  when the model is fair. The skeptic's goal, as it were, is to design payoff functions such that the wealth process grows quickly under the alternative, and thus rejection occurs sooner rather than later.
Inspired by a common idea in game-theoretic statistics (cf. [21, 26-28]), let us consider the following payoff function:
$$
S _ {t} \left(\widehat {Y} _ {t} ^ {0}, \widehat {Y} _ {t} ^ {1}\right) = 1 + \lambda_ {t} \left(\widehat {Y} _ {t} ^ {0} - \widehat {Y} _ {t} ^ {1}\right), \tag {2}
$$
where  $\lambda_{t}$  is predictable and lies in  $[-1, 1]$  to ensure that  $S_{t}(\widehat{Y}_{t}^{0}, \widehat{Y}_{t}^{1}) \geq 0$ . Note that for  $P \in H_{0}$ ,  $\mathbb{E}_{P}[\mathcal{K}_{t} | \mathcal{F}_{t-1}] = \mathcal{K}_{t-1} \mathbb{E}_{P}[1 + \lambda_{t} (\widehat{Y}_{t}^{0} - \widehat{Y}_{t}^{1}) | \mathcal{F}_{t-1}] = \mathcal{K}_{t-1}$ , so  $(\mathcal{K}_{t})_{t \geq 1}$  is a nonnegative  $P$ -martingale. Rejecting when  $\mathcal{K}_{t} > 1 / \alpha$  thus gives to a valid level- $\alpha$  sequential test as described above. We will select  $\lambda_{t}$  using Online Newton Step (ONS) [44, 45], which ensures exponential growth of the wealth process under the alternative. Figure 1 illustrates the behavior of the wealth process under various hypotheses when using ONS to choose  $\lambda_{t}$ . As the difference between the means  $\Delta = |\mu_{0} - \mu_{1}|$
Algorithm 1 Testing group fairness by betting
Input:  $\alpha \in (0,1)$
$$
\mathcal {K} _ {0} \gets 1
$$
for  $t = 1,2,\ldots ,\tau$  do
$\# \tau$  may not be known in advance
Perhaps receive audits  $\widehat{Y}_t^0$  and  $\widehat{Y}_t^{1}$
Construct payoff  $S_{t}$  (e.g., (2), (16), or (7))
Update  $\mathcal{K}_t\gets \mathcal{K}_{t - 1}\cdot S_t$
If  $\mathcal{K}_t\geq 1 / \alpha$  then stop and reject the null
if the null has not been rejected then
Draw  $U\sim \mathrm{Unif}(0,1)$  , reject if  $\kappa_{\tau}\geq U / \alpha$
Figure 1: Left: Description of main algorithm. Right: The wealth process  $(\mathcal{K}_t)$  under different conditions on the means. Observations follow a Bernoulli distribution for a given mean. As  $\Delta = |\mu_0 - \mu_1|$  increases, the wealth grows more quickly. If a test rejects, the procedure ends, hence the plateauing of the blue and purple lines. For  $\Delta = 0$ , the wealth fluctuates around 1 and the test never rejects. Shaded regions indicated the standard deviation after 100 trials.
increases, the wealth grows more quickly which leads to faster rejection of the null. To define ONS, let  $g_{t} = \hat{Y}_{t}^{0} - \hat{Y}_{t}^{1}$  and initialize  $\lambda_{1} = 0$ . For all  $t \geq 1$ , recursively define
$$
\lambda_ {t} = \left(\left(\frac {2}{2 - \ln (3)} \frac {z _ {t - 1}}{1 + \sum_ {i = 1} ^ {t - 1} z _ {i} ^ {2}} - \lambda_ {t - 1}\right) \wedge 1 / 2\right) \vee - 1 / 2, \text {w h e r e} z _ {i} = \frac {g _ {i}}{1 - \lambda_ {i} g _ {i}}. \tag {3}
$$
The machinery just introduced is sufficient to define a level- $\alpha$  sequential test. However, we add a final ingredient to increase the power of this procedure. Motivated by the randomized Ville's inequality mentioned in Section 2, we introduce one final step: If we stop the procedure at some stopping time  $\tau$  but have not yet rejected the null (because, for instance, our budget ran out), we can check if  $\mathcal{K}_{\tau} \geq U / \alpha$ , where  $U$  is uniform on  $[0,1]$  and drawn independently from everything observed so far. We emphasize that the final step can only be performed once. Thus, if it is possible that more data will be collected in the future, we advise waiting. We summarize the process in Algorithm 1.
The following proposition gives a bound on the expected stopping time of this sequential test under the alternative. We note that Shekhar and Ramdas [28] also provide a bound on the expected stopping time of a multivariate betting-style sequential test. However, due to the generality of their setting, their result is defined in terms of quantities which are difficult to analyze. We thus provide a more direct analysis specific to the difference of univariate means. The proof may be found in Appendix C.1.
Proposition 1. Algorithm 1 with input  $\alpha \in (0,1)$  and betting strategy (2) is a level- $\alpha$  sequential test with asymptotic power one. Moreover, letting  $\Delta = |\mu_0 - \mu_1|$ , under the alternative the expected stopping time  $\tau$  obeys
$$
\mathbb {E} [ \tau ] \lesssim \frac {1}{\Delta^ {2}} \log \left(\frac {1}{\Delta^ {2} \alpha}\right). \tag {4}
$$
It is possible to demonstrate a lower bound of  $\mathbb{E}[\tau] \gtrsim \sigma^2 \log(1 / \alpha) / \Delta^2$  where  $\sigma^2 = \mathbb{E}[(\hat{Y}^0 - \hat{Y}^1)^2]$  [28, Prop. 2]. Since  $\sigma^2 \gtrsim 1$  in the worst case, our result is optimal up to a factor of  $\log(1 / \Delta^2)$ .
3.2 Time-varying data collection policies
Here we extend the setting to allow for time-dependent data collection policies. This is motivated by the fact that organizations are often collecting data for various purposes and must therefore evaluate fairness on data which is not necessarily representative of the population at large. We also allow the data collection policies to differ by group. Mathematically, for  $b \in \{0,1\}$  let  $\pi_t^b (\cdot) = \pi_t(\cdot |\xi_b)$  denote the randomized policies which determine the probability with which a set of covariates are